PowerShell | Aidan Finn, IT Pro

Ignite 2016 – Discover What’s New In Windows Server 2016 Virtualization

This post is a collection of my notes from the Ben Armstrong’s (Principal Program Manager Lead in Hyper-V) session (original here) on the features of WS2016 Hyper-V. The session is an overview of the features that are new, why they’re there, and what they do. There’s no deep-dives.

A Summary of New Features

Here is a summary of what was introduced in the last 2 versions of Hyper-V. A lot of this stuff still cannot be found in vSphere.

And we can compare that with what’s new in WS2016 Hyper-V (in blue at the bottom). There’s as much new stuff in this 1 release as there were in the last 2!

Security

The first area that Ben will cover is security. The number of attack vectors is up, attacks are on the rise, and the sophistication of those attacks is increasing. Microsoft wants Windows Server to be the best platform. Cloud is a big deal for customers – some are worried about industry and government regulations preventing adoption of the cloud. Microsoft wants to fix that with WS2016.

Shielded Virtual Machines

Two basic concepts:

A VM can only run on a trusted & healthy host – a rogue admin/attacker cannot start the VM elsewhere. A highly secured Host Guardian Service must authorize the hosts.
A VM is encrypted by the customer/tenant using BitLocker – a rogue admin/attacker/government agency cannot inspect the VM’s contents by mounting the disk(s).

There are levels of shielding, so it’s not an all or nothing.

Key Storage Drive for Generation 1 VMs

Shielding, as above, required Generation 2 VMs. You can also offer some security for Generation 1 virtual machines: Key Storage Drive. Not as secure as shielded virtual machines or virtual TPM, but it does give us a safe way to use BitLocker inside a Generation 1 virtual machine – required for older applications that depend on older operating systems (older OSs cannot be used in Generation 2 virtual machines).

Virtual Secure Mode (VSM)

We also have Guest Virtual Secure Mode:

Credential Guard: protecting ID against pass-the-hash by hiding LSASS in a secured VM (called VSM) … in a VM with a Windows 10 or Windows Server 2016 guest OS! Malware running with admin rights cannot steal your credentials in a VM.
Device Guard: Protect the critical kernel parts of the guest OS against rogue s/w, again, by hiding them in a VSM in a Windows 10 or Windows Server 2016 guest OS.

Secure Boot for Linux Guests

Secure boot was already there for Windows in Generation 2 virtual machines. It’s now there for Linux guest OSs, protecting the boot loader and kernel against root kits.

Host Resource Protection (HRP)

Ben hopes you never see this next feature in action in the field This is because Host Resource Protection is there to protect hosts/VMs from a DOS attack against a host by someone inside a VM. The scenario: you have an online application running in a VM. An attacker compromises the application (example: SQL injection) and gets into the guest OS of the VM. They’re isolated from other VMs by the hypervisor and hardware/DEP, so they attack the host using DOS, and consume resources.

A new feature, from Azure, called HRP will determine that the VM is aggressively using resources using certain patterns, and start to starve it of resources, thus slowing down the DOS attack to the point of being pointless. This feature will be of particular interest to:

Companies hosting external facing services on Hyper-V/Windows Azure Pack/Azure Stack
Hosting companies using Hyper-V/Windows Azure Pack/Azure Stack

This is another great example of on-prem customers getting the benefits of Azure, even if they don’t use Azure. Microsoft developed this solution to protect against the many unsuccessful DOS attacks from Azure VMs, and we get it for free for our on-prem or hosted Hyper-V hosts. If you see this happening, the status of the VM will switch to Host Resource Protection.

Security Demos

Ben starts with virtual TPM. The Windows 10 VM has a virtual TPM enabled and we see that the C: drive is encrypted. He shuts down the VM to show us the TPM settings of the VM. We can optionally encrypt the state and live migration traffic of the VM – that means a VM is encrypted at rest and in transit. There is a “performance impact” for this optional protection, which is why it’s not on by default. Ben also enables shielding – and he loses console access to the VM – the only way to connect to the machine is to remote desktop/SSH to it.

Note: if he was running the full host guardian service (HGS) infrastructure then he would have had no control over shielding as a normal admin – only the HGS admins would have had control. And even the HGS admins have no control over BitLocker.

He switches to a Generation 1 virtual machine with Key Storage Drive enabled. BitLocker is running. In the VM settings (Generation 1) we see Security > Key Storage Drive Enabled. Under the hood, an extra virtual hard disk is attached to the VM (not visible in the normal storage controller settings, but visible in Disk Management in the guest OS). It’s a small 41 MB NTFS volume. The BitLocker keys are stored there instead of a TPM – virtual TPM is only in Generation 2, but it’s using the same sorts of tech/encryption/methods to secure the contents in the Key Storage Drive, but it cannot be as secure as virtual TPM, but it is better than not having BitLocker. Microsoft can make the same promises with data at rest encryption for Generation 1 VMs, but it’s still not as good as a Generation 2 VM with vTPM or even a shielded VM (requires Generation 2).

Availability

The next section is all about keeping services up and running in Hyper-V, whether it’s caused by upgrades or infrastructure issues. Everyone has outages and Microsoft wants to reduce the impact of these. Microsoft studied the common causes, and started to tackle them in WS2016

Cluster OS Rolling Upgrades

Microsoft is planning 2-3 updates per year for Nano Server, plus there’ll be other OS upgrades in the future. You cannot upgrade a cluster node. And in the past we could only do cluster-cluster migrations to adopt new versions of Windows Server/Hyper-V. Now, we can:

Remove cluster node 1
Rebuild cluster node 1 with the new version of Windows Server/Hyper-V
Add cluster node 1 to the old cluster – the cluster runs happily in mixed-mode for a short period of time (weeks), with failover and Live Migration between the old/new OS versions.
Repeat steps 1-3 until all nodes are up to date
Upgrade the cluster functional level – Update-ClusterFunctionalLevel (see below for “Emulex incident”)
Upgrade the VMs’ version level

Zero VM downtime, zero new hardware – 2 node cluster, all the way to a 64 node cluster.

If you have System Center:

Upgrade to SCVMM 2016.
Let it orchestrate the cluster upgrade (above)

Supports starts with WS2012 R2 to WS2016. Re-read that statement: there is no support for W2008/W2008 R2/WS2012. Re-read that last statement. No need for any questions now

To avoid an “Emulex incident” (you upgrade your hosts – and a driver/firmware fails even though it is certified, and the vendor is going to take 9 months to fix the issue) then you can actually:

Do the node upgrades.
Delay the upgrade to the cluster functional level for a week or two
Test your hosts/cluster for driver/firmware stability
Rollback the cluster nodes to the older OS if there is an issue –> only possible if the cluster functional level is on the older version.

And there’s no downtime because it’s all leveraging Live Migration.

Virtual Machine Upgrades

This was done automatically when you moved a VM from version X to version X+1. Now you control it (for the above to work). Version 8 is WS2016 host support.

Failover Clustering

Microsoft identified two top causes of outages in customer environments:

Brief storage “outages” – crashing the guest OS of a VM when an IO failed. In WS2016, when an IO fails, the VM is put in a paused-critical state (for up to 24 hours, by default). The VM will resume as soon as the storage resumes.
Transient network errors – clustered hosts being isolated causing unnecessary VM failover (reboot), even if the VM was still on the network. A very common 30 seconds network outage will cause a Hyper-V cluster to panic up to and including WS2012 R2 – attempted failovers on every node and/or quorum craziness! That’s fixed in WS2016 – the VMs will stay on the host (in an unmonitored state) if they are still networked (see network protection from WS2012 R2). Clustering will wait (by default) for 4 minutes before doing a failover of that VM. If a host glitches 3 times in an hour it will be automatically quarantined, after resuming from the 3rd glitch, (VMs are then live migrated to other nodes) for 2 hours, allowing operator inspection.

Guest Clustering with Shared VHDX

Version 1 of this in WS2012 R2 was limited – supported guest clusters but we couldn’t do Live Migration, replication, or backup of the VMs/shared VHDX files. Nice idea, but it couldn’t really be used in production (it was supported, but functionally incomplete) instead of virtual fibre channel or guest iSCSI.

WS2016 has a new abstracted form of Shared VHDX – it’s even a new file format. It supports:

Backup of the VMs at the host level
Online resizing
Hyper-V Replica (which should lead to ASR support) – if the workload is important enough to cluster, then it’s important enough to replicate for DR!

One feature that does not work (yet) is Storage Live Migration. Checkpoint can be done “if you know what you are doing” – be careful!!!

Replica Support for Hot-Add VHDX

We could hot-add a VHDX file to a VM, but we could not add that to replication if the VM was already being replicated. We had to re-replicate the VM! That changes in WS2016, thanks to the concept of replica sets. A new VHDX is added to a “not-replicated” set and we can move it to the replicated set for that VM.

Hot-Add Remove VM Components

We can hot-add and hot-remove vNICs to/from running VMs. Generation 2 VMs only, with any supported Windows or Linux guest OS.

We can also hot-add or hot-remove RAM to/from a VM, assuming:

There is free RAM on the host to add to the VM
There is unused RAM in the VM to remove from the VM

This is great for those VMs that cannot use Dynamic Memory:

No support by the workload
A large RAM VM that will benefit from guest-aware NUMA

A nice GUI side-effect is that guest OS memory demand is now reported in Hyper-V Manager for all VMs.

Production Checkpoints

Referring to what used to be called (Hyper-V) snapshots, but were renamed to checkpoints to stop dumb people from getting confused with SAN and VSS snapshots – yes, people really are that stupid – I’ve met them.

Checkpoints (what are now called Standard Checkpoints) were not supported by many applications in a guest OS because they lead to application inconsistency. WS2016 adds a new default checkpoint type called a Production Checkpoint. This basically uses backup technology (and IT IS STILL NOT A BACKUP!) to create an application consistent checkpoint of a VM. If you apply (restore) the checkpoint the VM:

The VM will not boot up automatically
The VM will boot up as if it was restoring from a backup (hey dumbass, checkpoints are STILL NOT A BACKUP!)

For the stupid people, if you want to backup VMs, use a backup product. Altaro goes from free to quite affordable. Veeam is excellent. And Azure Backup Server gives you OPEX based local backup plus cloud storage for the price of just the cloud component. And there are many other BACKUP solutions for Hyper-V.

Now with production checkpoints, MSFT is OK with you using checkpoints with production workloads …. BUT NOT FOR BACKUP!

Demos

Ben does some demos of the above. His demo rig is based on nested virtualization. He comments that:

The impact of CPU/RAM is negligible
There is around a 25% impact on storage IO

Storage

The foundation of virtualization/cloud that makes or breaks a deployment.

Storage Quality of Service (QOS)

We had a basic system in WS2012 R2:

Set max IOPS rules per VM
Set min IOPS alerts per VM that were damned hard to get info from (WMI)

And virtually no-one used the system. Now we get storage QoS that’s trickled down from Azure.

In WS2016:

We can set reserves (that are applied) and limits on IOPS
Available for Scale-Out File Server and block storage (via CSV)
Metrics rules for VHD, VM, host, volume
Rules for VHD, VM, service, or tenant
Distributed rule application – fair usage, managed at storage level (applied in partnership by the host)
PoSH management in WS2016, and SCVMM/SCOM GUI

You can do single-instance or multi-instance policies:

Single-instance: IOPS are shared by a set of VMs, e.g. a service or a cluster, or this department only gets 20,000 IOPS.
Multi-instance: the same rule is applied to a group of VMs, the same rule for a large set of VMs, e.g. Azure guarantees at least X IOPS to each Standard storage VHD.

Discrete Device Assignment – NVME Storage

DDA allows a virtual machine to connect directly to a device. An example is a VM connects directly to extremely fast NVME flash storage.

Note: we lose Live Migration and checkpoints when we use DDA with a VM.

Evolving Hyper-V Backup

Lots of work done here. WS2016 has it’s only block change tracking (Resilient Change Tracking) so we don’t need a buggy 3rd party filter driver running in the kernel of the host to do incremental backups of Hyper-V VMs. This should speed up the support of new Hyper-V versions by the backup vendors (except for you-know-who-yellow-box-backup-to-tape-vendor-X, obviously!).

Large clusters had scalability problems with backup. VSS dependencies have been lessened to allow reliable backups of 64 node clusters.

Microsoft has also removed the need for hardware VSS snapshots (a big source of bugs), but you can still make use of hardware features that a SAN can offer.

ReFS Accelerated VHDX Operations

Re-FS is the preferred file system for storing VMs in WS2016. ReFS works using metadata which links to data blocks. This abstraction allows very fast operations:

Fixed VHD/X creation (seconds instead of hours)
Dynamic VHD/X expansion
Checkpoint merge, which impacts VM backup

Note, you’ll have to reformat WS2012 R2 ReFS to get the new version of ReFS.

Graphics

A lot of people use Hyper-V (directly or in Azure) for RDS/Citrix.

RemoteFX Improvements

The AVC444 thing is a lossless codec – lossless 3D rendering, apparently … that’s gobbledegook to me.

DDA Features and GPU Capabilities

We can also use DDA to connect VMs directly to CPUs … this is what the Azure N-Series VMs are doing with high-end NVIDIA GFX cards.

DirectX, OpenGL, OpenCL, CUDA
Guest OS: Server 2012 R2, Server 2016, Windows 10, Linux

The h/w requirements are very specific and detailed. For example, I have a laptop that I can do RemoteFX with, but I cannot use for DDA (SRIOV not supported on my machine).

Headless Virtual Machine

A VM can be booted without display devices. Reduces the memory footprint, and simulates a headless server.

Operational Efficiency

Once again, Microsoft is improving the administration experience.

PowerShell Direct

You can now to remote PowerShell into a VM via the VMbus on the host – this means you do not need any network access or domain join. You can do either:

Enter-PSSession for an interactive session
Invoke-Command for a once-off instruction

Supports:

Host: Windows 10/WS2016
Guest: Windows 10/WS2016

You do need credentials for the guest OS, and you need to do it via the host, so it is secure.

This is one of Ben’s favourite WS2016 features – I know he uses it a lot to build demo rigs and during demos. I love it too for the same reasons.

PowerShell Direct – JEA and Sessions

The following are extensions of PowerShell Direct and PowerShell remoting:

Just Enough Administration (JEA): An admin has no rights with their normal account to a remote server. They use a JEA config when connecting to the server that grants them just enough rights to do their work. Their elevated rights are limited to that machine via a temporary user that is deleted when their session ends. Really limits what malware/attacker can target.
Justin-Time Administration (JITA): An admin can request rights for a short amount of time from MIM. They must enter a justification, and company can enforce management approval in the process.

vNIC Identification

Name the vNICs and make that name visible in the guest OS. Really useful for VMs with more than 1 vNIC because Hyper-V does not have consistent device naming.

Hyper-V Manager Improvements

Yes, it’s the same MMC-based Hyper-V Manager that we got in W2008, but with more bells and whistles.

Support for alternative credentials
Connect to a host IP address
Connect via WinRM
Support for high-DPI monitors
Manage WS2012, WS2012 R2 and WS2016 from one HVM – HVM in Win10 Anniversary Update (The big Redstone 1 update in Summer 2016) has this functionality.

VM Servicing

MS found that the vast majority of customers never updated the Integration services/components (ICs) in the guest OS of VMs. It was a horrible manual process – or one that was painful to automate. So customers ran with older/buggy versions of ICs, and VMs often lacked features that the host supported!

ICs are updated in the guest OS via Windows Update on WS2016. Problem sorted, assuming proper testing and correct packaging!

MSFT plans to release IC updates via Windows Update to WS2012 R2 in a month, preparing those VMs for migration to WS2016. Nice!

Core Platform

Ben was running out of time here!

Delivering the Best Hyper-V Host Ever

This was the Nano Server push. Honestly – I’m not sold. Too difficult to troubleshoot and a nightmare to deploy without SCVMM.

I do use Nano in the lab. Later, Ben does a demo. I’d not seen VM status in the Nano console before, which Ben shows – the only time I’ve used the console is to verify network settings that I set remotely using PoSH There is also an ability to delete a virtual switch on the console.

Nested Virtualization

Yay! Ben admits that nested virtualization was done for Hyper-V Containers on Azure, but we people requiring labs or training environments can now run multiple working hosts & clusters on a single machine!

VM Configuration File

Short story: it’s binary instead of XML, improving performance on dense hosts. Two files:

.VMCX: Configuration
.VMRS: Run state

Power Management

Client Hyper-V was impacted badly by Windows 8 era power management features like Connected Standby. That included Surface devices. That’s sorted now.

Development Stuff

This looks like a seed for the future (and I like the idea of what it might lead to, and I won’t say what that might be!). There is now a single WMI (Root\HyperVCluster\v2) view of the entire Hyper-V cluster – you see a cluster as one big Hyper-V server. It really doesn’t do much now.

And there’s also something new called Hyper-V sockets for Microsoft partners to develop on. An extension of the Windows Socket API for “fast, efficient communication between the host and the guest”.

Scale Limits

The numbers are “Top Gear stats” but, according to a session earlier in the week, these are driven by Azure (Hyper-V’s biggest customer). Ben says that the numbers are nuts and we normals won’t ever have this hardware, but Azure came to Hyper-V and asked for bigger numbers for “massive scale”. Apparently some customers want massive super computer scale “for a few months” and Azure wants to give them an OPEX offering so those customers don’t need to buy that h/w.

Note Ben highlights a typo in max RAM per VM: it should say 12 TB max for a VM … what’s 4 TB between friends?!?!

Ben wraps up with a few demos.

RunAsRadio Podcast – Hyper-V in Server 2016

I recently recorded an episode of the RunAsRadio podcast with Richard Campbell on the topic of Windows Server 2016 (WS2016) Hyper-V. We covered a number of areas, including containers, nested virtualization, networking, security, and PowerShell.

Technorati Tags: Windows Server 2016,Hyper-V,Virtualisation,PowerShell,Security,Networking,Failover Clustering

Optimize Hyper-V VM Placement To Match CSV Ownership

This post shares a PowerShell script to automatically live migrate clustered Hyper-V virtual machines to the host that owns the CSV that the VM is stored on. The example below should work nicely with a 2-node cluster, such as a cluster-in-a-box.

For lots of reasons, you get the best performance for VMs on a Hyper-V cluster if:

Host X owns CSV Y AND
The VMs that are stored on CSV Y are running on Host X.

This continues into WS2016, as we’ve seen by analysing the performance enhancements of ReFS for VHDX operations. In summary, the ODX-like enhancements work best when the CSV and VM placement are identical as above.

I wrote a script, with little bits taken from several places (scripting is the art of copy & paste), to analyse a cluster and then move virtual machines to the best location. The method of the script is:

Move CSV ownership to what you have architected.
Locate the VMs that need to move.
Order that list of VMs based on RAM. I want to move the smallest VMs first in case there is memory contention.
Live migrate VMs based on that ordered list.

What’s missing? Error handling 🙂

What do you need to do?

You need to add variables for your CSVs and hosts.
Modify/add lines to move CSV ownership to the required hosts.
Balance the deployment of your VMs across your CSVs.

Here’s the script. I doubt the code is optimal, but it works. Note that the Live Migration command (Move-ClusterVirtualMachineRole) has been commented out so you can see what the script will do without it actually doing anything to your VM placement. Feel free to use, modify, etc.

#List your CSVs 
$CSV1 = "CSV1" 
$CSV2 = "CSV2"

#List your hosts 
$CSV1Node = "Host01" 
$CSV2Node = "Host02"

function ListVMs () 
{ 
    Write-Host "`n`n`n`n`n`nAnalysing the cluster $Cluster ..."

    $Cluster = Get-Cluster 
    $AllCSV = Get-ClusterSharedVolume -Cluster $Cluster | Sort-Object Name

    $VMMigrationList = @()

    ForEach ($CSV in $AllCSV) 
    { 
        $CSVVolumeInfo = $CSV | Select -Expand SharedVolumeInfo 
        $CSVPath = ($CSVVolumeInfo).FriendlyVolumeName

        $FixedCSVPath = $CSVPath -replace '\\', '\\'

        #Get the VMs where VM placement doesn't match CSV ownership
        $VMsToMove = Get-ClusterGroup | ? {($_.GroupType –eq 'VirtualMachine') -and ( $_.OwnerNode -ne $CSV.OWnernode.Name)} | Get-VM | Where-object {($_.path -match $FixedCSVPath)} 

        #Build up a list of VMs including their memory size 
        ForEach ($VM in $VMsToMove) 
        { 
            $VMRAM = (Get-VM -ComputerName $VM.ComputerName -Name $VM.Name).MemoryAssigned

            $VMMigrationList += ,@($VM.Name, $CSV.OWnernode.Name, $VMRAM) 
        }

    }

    #Order the VMs based on memory size, ascending 
    $VMMigrationList = $VMMigrationList | sort-object @{Expression={$_[2]}; Ascending=$true}

    Return $VMMigrationList 
}

function MoveVM ($TheVMs) 
{

    foreach ($VM in $TheVMs) 
        { 
        $VMName = $VM[0] 
        $VMDestination = $VM[1] 
        Write-Host "`nMove $VMName to $VMDestination" 
        #Move-ClusterVirtualMachineRole -Name $VMName -Node $VMDestination -MigrationType Live 
        }

}

cls

#Configure which node will own wich CSV 
Move-ClusterSharedVolume -Name $CSV1 -Node $CSV1Node | Out-Null 
Move-ClusterSharedVolume -Name $CSV2 -Node $CSV2Node | Out-Null

$SortedVMs = @{}

#Get a sorted list of VMs, ordered by assign memory 
$SortedVMs = ListVMs

#Live Migrate the VMs, so that their host is also their CSV owner 
MoveVM $SortedVMs

Possible improvements:

My ListVMs algorithm probably can be improved.
The Live Migration piece also can be improved. It only does 1 VM at a time, but you could implement parallelism using jobs.
Quick Migration should be used for non-running VMs. I haven’t handles that situation.
You could opt to use Quick Migration for low priority VMs – if that’s your policy.
The script could be modified to start using parameters, e.g. Analyse (not move), QuickMigrateLow, QuickMigrate (instead of Live Migrate), etc.

Technorati Tags: Hyper-V,PowerShell,Scripting,Failover Clustering,Storage,Storage Spaces

Playing with WS2016 Hyper-V – Nested Virtualization, Nano, SET, and PowerShell Direct

I have deployed Technical Preview 5 (TP5) of Windows Server 2016 (WS2016) to most of the hardware in my lab. One of the machines, a rather old DL380 G6, is set up as a standalone host. I’m managing it using Remote Server Administration Toolkit (RSAT) for Windows 10 (another VM).

I enabled Hyper-V on that host. I then deployed a 4 x Generation 2 VMs using Nano Server (domain pre-joined using .djoin files) – this keeps the footprint tiny and the boot times are crazy fast.

Hyper-V is enabled in the Nano VMs – thanks to the addition of nested virtualization. I’ve also clustered these machines. Networking-wise, I have given each VM 2 x vNICs, each with MAC spoofing (for nested VMs) and NIC teaming enabled.

I launched PowerShell ISE then used Enter-PSSession to connect to the host from the admin PC. And from the host, I used Enter-PSSession -VMName to use PowerShell Direct to get into each VM – this gives me connectivity without depending on the network. That’s because I wanted to deploy Switch Embedded Teaming (SET) and provision networking in the Nano VMs. This script configure the VMs each with 3 vNICs for the management OS, connected to the vSwitch that uses both of the Nano VMs vNICs as teamed uplinks:

$idx = 54

New-VMSwitch -Name External -NetAdapterName "Ethernet","Ethernet 2" -EnableEmbeddedTeaming $true -AllowManagementOS $false

Add-VMNetworkAdapter -ManagementOS -Name "Management" -SwitchName External
Add-VMNetworkAdapter -ManagementOS -Name "SMB1" -SwitchName External
Add-VMNetworkAdapter -ManagementOS -Name "SMB2" -SwitchName External

Sleep 10

New-NetIPAddress -InterfaceAlias "vEthernet (Management)" -IPAddress 172.16.2.$idx -PrefixLength 16  -DefaultGateway 172.16.1.1
Set-DnsClientServerAddress -InterfaceAlias "vEthernet (Management)" -ServerAddresses "172.16.1.40"

New-NetIPAddress -InterfaceAlias "vEthernet (SMB1)" -IPAddress 192.168.3.$idx -PrefixLength 24
New-NetIPAddress -InterfaceAlias "vEthernet (SMB2)" -IPAddress 192.168.4.$idx -PrefixLength 24

Note: there’s no mention of RDMA because I’m working in a non-RDMA scenario – a test/demo lab. Oh yes; you can learn Hyper-V, Live Migration, Failover Clustering, etc on your single PC now!

And in no time, I had myself a new Hyper-V cluster with a tiny physical footprint, thanks to 4 new features in WS2016.

Technorati Tags: Windows Server 2016,Hyper-V,PowerShell

Cannot Bind Parameter ‘ForegroundColor’ Error When Creating Nano Server Image

You get the following error when running New-NanoServerImage in PowerShell ISE to create a new Windows Server 2016 (WS2016) Nano Server image:

Write-W2VError : Cannot bind parameter ‘ForegroundColor’. Cannot convert the “#FFFF0000” value of type
“System.Windows.Media.Color” to type “System.ConsoleColor”.

The fix (during TP5) is to not use PowerShell ISE. Use an elevated PowerShell prompt instead. The reasoning is explained here by “daviwil”.

Russinovich on Hyper-V Containers

We’ve known since Ignite 2015 that Microsoft was going to have two kinds of containers in Windows Server 2016 (WS2016):

Windows Server Containers: Providing OS and resource virtualization and isolation.
Hyper-V Containers: The hypervisor adds security isolation to machine & resource isolation.

Beyond that general description, we knew almost nothing about Hyper-V Containers, other than expect them in preview during Q4 of 2015 – Technical Preview 4 (TPv4), and that it is the primary motivation for Microsoft to give us nested virtualization.

That also means that nested virtualization will come to Windows Server 2016 Hyper-V in TPv4.

We have remained in the dark since then, but Mark Russinovich appeared on Microsoft Mechanics (a YouTube webcast by Microsoft) and he explained a little more about Hyper-V Containers and he also did a short demo.

Some background first. Normally, a machine has a single user mode running on top of kernel mode. This is what restricts us to the “one app per OS” best practice/requirement, depending on the app. When you enable Containers on WS2016, an enlightenment in the kernel allows multiple user modes. This gives us isolation:

Namespace isolation: Each container sees it’s own file system and registry (the hives in the containers hosted files).
Resource isolation: How much process, memory, and CPU a container can use.

Kernel mode is already running when you start a new container, which improves the time to start up a container, and thus it’s service(s). This is great for deploying and scaling out apps because a containerised app can be deployed and started in seconds from a container image with no long term commitment, versus minutes for an app in a virtual machine with a longer term commitment.

But Russinovich goes on to say that while containers are great for some things that Microsoft wants to do in Azure, they also have to host “hostile multi-tenant code” – code uploaded by Microsoft customers that Microsoft cannot trust and that could be harmful or risky to other tenants. Windows Server Containers, like their Linux container cousins, do not provide security isolation.

In the past, Microsoft has placed such code into Hyper-V (Azure) virtual machines, but that comes with a management and direct cost overhead. Ideally, Microsoft wants to use lightweight containers with the security isolation of machine virtualization. And this is why Microsoft created Hyper-V Containers.

Hyper-V provides excellent security isolation (far fewer vulnerabilities found than vSphere) that leverages hardware isolation. DEP is a requirement. WS2016 is introducing IOMMU support, VSM, and Shielded Virtual Machines, with a newly hardened hypervisor architecture.

Hyper-V containers use the exact same code or container images as Windows Server Containers.That makes your code interchangeable – Russinovich shows a Windows Server Container being switched into a Hyper-V container by using PowerShell to change the run type (container attribute RuntimeType).

The big difference between the two types, other than the presence f Hyper-V, is that Hyper-V Containers get their own optimized instance of Windows running inside of them, as the host for the single container that they run.

The Hyper-V Container is not a virtual machine – Russinovich demonstrates this by searching for VMs with Get-VM. It is a container, and is manageable by the same commands as a Windows Server Container.

In his demos he switches a Windows Server Container to a Hyper-V Container by running:

Set-Container -Name <Container Name> -RuntimeType HyperV

And then he queries the container with:

Get-Container -Name <Container Name> | fl Name, State, RuntimeType

So the images and the commands are common across Hyper-V Containers and Windows Server Containers. Excellent.

It looked to me that starting this Hyper-V Container is a slower operation than starting a Windows Server Container. That would make sense because the Hyper-V container requires it’s own operating system.

I’m guessing that Hyper-V Containers either require or work best with Nano Server. And you can see why nested virtualization is required. A physical host will run many VM hosts. A VM host might need to run Hyper-V containers – therefore the VM Host needs to run Hyper-V and must have virtualized VT-x instructions.

Russinovich demonstrates the security isolation. Earlier in the video he queries the processes running in a Windows Server Container. There is a single CSRSS process in the container. He shows that this process instance is also visible on the VM host (same process ID). He then does the same test with a Hyper-V Container – the container’s CSRSS process is not visible on the VM host because it is contained and isolated by the child boundary of Hyper-V.

What about Azure? Microsoft wants Azure to be the best place to run containers – he didn’t limit this statement to Windows Server or Hyper-V, because Microsoft wants you to run Linux containers in Azure too. Microsoft announced the Azure Container Service, with investments in Docker and Mesospehere for deployment and automation of Linux, Windows Server, and Hyper-V containers. Russinovich mentions that Azure Automation and Machine Learning will leverage containers – this makes sense because it will allow Microsoft to scale out services very quickly, in a secure manner, but with less resource and management overhead.

That was a good video, and I recommend that you watch it.

Technorati Tags: Containers,Windows Server 2016,Hyper-V,Virtualisation,PowerShell,Azure

Create a WS2016 Nano Server Hyper-V VM

Setting up a Nano Server VM requires running some PowerShell. The instructions that I found out there aren’t that clear for a non-PowerShell guru , are wrong, or are incomplete. So let me clear up everything by showing you exactly what I am using to deploy Nano Server as a Windows Server 2016 (TPv3/Technical Preview 3) Hyper-V virtual machine.

Note: The process will probably change after I published this post.

Step 1 – Make Folders

Create three folders on a computer with a fast disk. Note that I’ll use C: but maybe you should use a D: or something.

C:\Nano
C:\Nano\Base
C:\Scripts

Step 2 – Copy Scripts

Mount the WS2016 ISO – let’s assume that it mounts as E:. Copy two scripts from E:\NanoServer from the ISO to C:\Scripts:

new-nanoserverimage.ps1
convert-windowsimage.ps1

Step 3 – Dot The Scripts

Note that I missed out on this step because I had never encountered this sort of thing before – I’m an advocate of PowerShell but I’m no guru! If you do not run this step, New-NanoServerImage.ps1 will do nothing at all and wreck your head for 3 hours (it did for me!).

Open a PowerShell window with elevated privileges. Navigate to C:\Scripts. Run the following:

. .\convert-windowsimage.ps1

I know – it looks funny. Enter it exactly as above. This appears to load the contained script as a cmdlet that can later be executed.

Do the same again for New-NanoServerVHD.ps1:

. .\new-nanoserverimage.ps1

Now we can build a new VHD with Nano Server pre-installed.

Step 4 – Create a VHD

You can now run New-NanoServerImage. Here’s what I ran:

New-NanoServerImage -MediaPath e:\ -BasePath C:\Nano\Base -TargetPath C:\Nano\Nano1 -GuestDrivers -ComputerName "Nano1" -DomainName "prev.internal" -EnableIPDisplayOnBoot -AdministratorPassword (convertto-securestring -string "AVerySecurePassPhrase" -asplaintext -force) -EnableRemoteManagementPort -Language EN-US

The above will prep a VHD with a VM called Nano1. I have configured the VM to join the prev.internal domain – note that this will require me to have suitable domain creds – a computer account is created in the domain. I enabled the Hyper-V guest drivers and allowed the IP of the VM to appear on the console. The VHD will be stored in C:\Nano\Nano1. Note that if this folder exists then the process will abort:

WARNING: The target directory already exists. If you want to rebuild this image, delete the directory first.
WARNING: Terminating due to an error. See log file at:
C:\Users\ADMINI~1.LAB\AppData\Local\Temp\2\New-NanoServerImage.log

Note that I had to specify EN-US because, at this time, my default region of EN-IE was not available:

WARNING: The ‘en-ie’ directory does not exist in the ‘Packages’ directory (‘g:\NanoServer\Packages’).
WARNING: Terminating due to an error. See log file at:
C:\Users\ADMINI~1.LAB\AppData\Local\Temp\2\New-NanoServerImage.log

I could have added other roles/packages to the VHD such as:

-Storage: For a SOFS cluster.
-Compute: To enable Hyper-V … useful when TPv4 (we guess) introduces guest virtualization.
-Clustering: To enable failover clustering in the VM.
-Defender: Adding security to the guest OS.

A minute or so later, a 439 MB was created in the newly created C:\Nano\Nano1.

Recreating a Nano Server VM

If you’re playing with Nano Server in a lab then you’ll create VMs with name reuse. If you do this with domain join then you might encounter a failure:

WARNING: Failed with 2224.
WARNING: Terminating due to an error. See log file at:
C:\Users\ADMINI~1.LAB\AppData\Local\Temp\2\New-NanoServerImage.log

Open the log and you’ll find:

Provisioning the computer…

Failed to provision [Nano1] in the domain [prev.internal]: 0x8b0.

It may be necessary to specify /REUSE when running

djoin.exe again with the same machine name.

Computer provisioning failed: 0x8b0.

The account already exists.

That’s one of those “ding-ding-ding aha!” moments. The computer account already exists in AD so delete the account and start over.

Creating Additional VMs

Once you have run the above process, C:\Nano\Base will be populated with files from the ISO (\NanoServer). This means that you can drop the -MediaPath flag and eject the ISO.

New-NanoServerImage -BasePath C:\Nano\Base -TargetPath C:\Nano\Nano2 -GuestDrivers -ComputerName "Nano2" -DomainName "prev.internal" -EnableIPDisplayOnBoot -AdministratorPassword (convertto-securestring -string "AVerySecurePassPhrase" -asplaintext -force) -EnableRemoteManagementPort -Language EN-US

Step 5 – Move the Computer Account

In AD, move the computer account for the new Nano server to the required OU so it get’s any requierd policies on the first boot – remember that this sucker has no UI so GPO and stuff like Desired State Configuration (DSC) will eventually be the best way to configure Nano Server.

Step 6 – Create a VM

The above process prepare a VHD for a Generation 1 virtual machine. Create a Generation 1 VM, and attach the VHD to the boot device. Connect to the VM and power it up. A couple of seconds will pass and a log in screen will appear:

Log in with your local admin or domain credentials and you’ll be greeted with the console. Note that I enabled the IP address to be displayed during the setup:

Step 7 – Manage the Nano Server VM

If you want to do some management work then you’ll need to:

Wait for the eventual remote management console that was quickly shown at Ignite 2015.
Use PowerShell remoting.
Use PowerShell Direct (new in WS2016).

If you have network access to the VM then you can use remoting:

Enter-PSSession -ComputerName Nano1 -Credential prev\administrator

Troubleshooting network issues with Nano Server can be a dog because there is no console that you can log into. However … you can use PowerShell Direct with no network access to the VM, via the Hyper-V guest OS integration components:

Enter-PSSession -VMName Nano1 -Credential prev\administrator

Tip: Most AD veterans start network troubleshooting with DNS – it’s nearly always the cause. In my lab, I have 3 domains, so 3 sets of DNS. My DHCP scope sets up on domain’s DNS server as the primary, and that can cause issues. Some PowerShell Direct to the VM with some Set-DnsClientServerAddress sorted things out.

Technorati Tags: Hyper-V,Windows Server 2016,Deployment,PowerShell

Microsoft News – 30 September 2015

Microsoft announced a lot of stuff at AzureCon last night so there’s lots of “launch” posts to describe the features. I also found a glut of 2012 R2 Hyper-V related KB articles & hotfixes from the last month or so.

Hyper-V

The uptime value of the Hyper-V Management console changes to the resume time of the last pause on Windows Server 2012 and Windows Server 2012 R2: The uptime value of the Hyper-V Management console changes to the resume time of the last pause instead of the time when backup completed
A virtual machine that is running on Windows Server 2012 R2 may not start: Not enough memory in the system to start the virtual machine ‘<Virtual Machine Name>’.
Non-queued commands are sent to disk controller on a Windows Server 2012 R2-based Hyper-V host server: A solution to a non-described issue – how weird!
Cluster service stops during the VSS backup in a Windows Server 2012 R2-based Hyper-V cluster: When multiple VSS backups are in progress, the cluster service may crash because of inconsistent state in the CSV VSS provider. Then, all virtual machines go offline because of the CSV failure.
Hyper-V storage migration failure and “ERROR_BAD_COMMAND” error in Windows Server 2012 R2 or Windows Server 2012: You get “Migration did not succeed. Not enough disk space at ‘\’. ” when doing Storage Live Migration to a CSV or Hyper-V Replica stops with ERROR_BAD_COMMAND.

Windows Server

Applications can’t perform I/Os in Windows Server 2012 R2: When a disk fails or isn’t responsive and a large number of I/O port transitions are pending, I/Os may be stalled because the storage adapter is paused and will be resumed after a time-out period as specified by the miniport.
Changed cluster properties revert to default values on cluster nodes that run Windows Server 2012 R2 or Windows Server 2008 SP2: ameSubnetDelay, SameSubnetThreshold, CrossSubnetDelay, or CrossSubnetThreshold properties revert to original values.
Cluster services go offline when there’s a connectivity issue in Windows Server 2012 R2 or Windows Server 2012: The cluster nodes in both datacenters lose quorum and fail when a connectivity issue occurs between datacenters.

Azure

Using Azure Automation to take action on Azure Alerts: Reduce human effort by automation solutions to recurring problems.
Announcing expanded Microsoft Azure support for financial services customers: Adding European Network and Information Security Agency (ENISA) Information Assurance Framework (IAF) and the Shared Assessments Program (formerly known as BITS Shared Assessments).
Microsoft ExpressRoute – Office 365, Azure Government Cloud, more partners, simplified billing: Place equipment (e.g. SANs) in Microsoft “Meet-Me” locations with low latency connections (1-2 ms) to Azure.
Azure Container Service – Now and the Future: Azure Container Service builds on our work with Docker and Mesosphere to create and manage scalable clusters of host machines onto which containerized applications can be deployed, orchestrated, and managed.
Making the cloud more secure: Azure Security Center—a new Azure service that gives you visibility and control of the security of your Azure resources without impeding agility, and helps you stay ahead of cyber threats even as they evolve.
Azure File Storage, now generally available: Shared storage for APPLICATION data. This is not for user files.
How to use Azure File storage with Windows: Here are the Linux instructions.
Azure Storage Improvements: A roundup, including Premium Storage improvements and region support of Import/Export.
Monitor your Azure RemoteApp environment with OpInsight – Part 3: Monitoring the usage of your Azure RemoteApp collection.

Office 365

Announcing general availability of ExpressRoute for Office 365: This direct connection offers customers more predictable network performance, an SLA for guaranteed availability and additional data privacy.
Office 365 release options: Why am I not getting the Office 365 Click-To-Run update from Office 365? Are you in the Standard Release or First Release update rings?

EMS

Microsoft Intune Company Portal for Android: A new version.

Technorati Tags: Hyper-V,Windows Server 2012 R2,Windows Server 2012,Backup,Failover Clustering,DR,Azure,Cloud,Cloud Computing,Hybrid Cloud,PowerShell,Scripting,Networking,Containers,Storage,Remote Desktop Services,Office,Office 365,EMS,Intune

Upgrade An Azure-Hosted Service By Moving A VIP To A New Cloud Service

Last Friday I talked about how you could reserve and manipulate cloud service VIPs. In this post I’m going to show you how to “upgrade” a service by moving to a new installation of that service running in a new cloud service – this can be done by moving the VIP of the original cloud service to the new cloud service.

Have you wondered how you will upgrade your WS2012 R2 VMs to WS2016 in Azure? The answer is that you won’t. You will have to migrate services to new VMs. Here’s a way to do that migration. This process will keep the original installation running while the new service is being built. Once ready, the VIP (the public IP of the original service) is migrated to the newer cloud service. If all goes well, you remove the old cloud service. If all sucks, you migrate the VIP back to the original cloud service.

In my lab I have two cloud services:

OldWeb: This runs a WS2012 R2 VM with IIS
NewWeb2016: This runs a WS2016 VM with IIS

Let’s say I have a site called http://www.joeelway.com. The A records for joeelway.com and http://www.joeelway.com will point to this VIP of the OldWeb cloud service; this is what allows a browser to connect to that site. If I don’t have a reserved VIP then I can create one easily enough with:

New-AzureReservedIP -ReservedIPName "WebsiteVIP" -Location "North Europe" -ServiceName "OldWeb"

This will reserve the existing IPv4 address that is used by OldWeb with the cloud service. This is a non-disruptive change that simply fixes the existing IP address with the cloud service. I can continue to browse to the website using the same VIP as when it was dynamic.

Now I can build up a new web application using the NewWeb2016 cloud service. This has zero impact on the OldWeb cloud service, running side-by-side but using a different (probably dynamic) VIP:

The A records for the joeelway.com domain continue to point at the reserved VIP for OldWeb, so users are still going to the old service.

And then we plan a switchover, with all of the necessary data copy/replication/synchronisation, change controls, reviews, communications, etc. How do I make the change? It’s simple; we run two cmdlets to change the reserved IP association.

The first cmdlet will remove the association of the reserved VIP from the OldWeb cloud service. This forces the old service to get a new dynamic VIP:

Remove-AzureReservedIPAssociation -ReservedIPName "WebsiteVIP" -ServiceName “OldWeb”

This cmdlet takes a few minutes to run so plan for the associated outage that will be caused. The A records for the joeelway.com domain continue to point at the reserved VIP, which is no longer associated with a service. If you browse to the VIP the connection will time out:

We want to avoid such a time out experience for the site’s users so we will very quickly associate the VIP with the new cloud service to minimise downtime (scripting is perfect for this!):

Set-AzureReservedIPAssociation -ReservedIPName "WebsiteVIP" -ServiceName "NewWeb2016"

The A records continue to resolve to the reserved VIP, and now the VIP is associated to the new cloud service:

If all goes well, you can decommission the old cloud service (VMs, etc), but you can leave them running for a little while as a rollback plan:

Remove the VIP association from the new cloud service
Set the VIP association with the old cloud service

You have to admit that, even if you are a PowerShell hater, this is a nice way to switch clients to a new version of a service.

Technorati Tags: Azure,Networking,PowerShell,Scripting

Configuring Windows Server Containers To Use DHCP Instead Of NAT

Read on if you want to learn how to connect Windows Server containers to an external virtual switch so that you don’t use NAT, and the containers actually talk directly to the LAN via DHCP assigned addresses. You’ll also see why a DHCP enabled container fails to get and address and ends up with a 169.254.x.x APIPA IPv4 configuration.

If you use Microsoft’s setup scripts for Windows Server 2016 (WS2016) Technical Preview 3 (TPv3), the default configuration for container networking is that each VM host will have virtual switch (in the VM), connected the VM’s vNIC. The virtual switch works in NAT mode, and uses a private network range to dynamically address containers that connect to the virtual switch. This set up requires each container to have NAT rules on the VM host so that external clients can connect to the services running in the containers. That … could be messy. In some terms, it could allow for huge network scalability (with tens of thousands of possible ports per VM host) but in others, it could be a nightmare to orchestrate.

What if you wanted your containers to talk directly on the LAN. In other words: no NAT. Yes, your containers can do this, and it’s known as a DHCP configuration – your containers are stateless so it’s pointless assigning them static IP addresses; instead the containers will get their addressing from DHCP services on the LAN.

Remember that there are two scripts that we can run to set up a VM host.

Method 1: You download New-ContainerHost.ps1 and run it. This downloads a bunch of stuff, creates a VM host, and then runs Install-ContainerHost.ps1. By default, this will configure the VM host with NAT networking.
Method 2: You create your own VM, download and run Install-ContainerHost.ps1. By default, you’ll get NAT networking.

But …

Install-ContainerHost.ps1 includes the option for a flag:

If you use method 2 then you could run Install-ContainerHost in the new VM host with the -UseDHCP flag set to $true; the behaviour of the script will change. By default it creates the VM host’s virtual switch in NAT mode. But enabling this flag creates an external virtual switch.

In my lab, I like to create my VM hosts using New-ContainerHost because it’s very quick (thanks to the use of differencing disks) and automates the entire setup. But New-ContainerHost doesn’t include the option for UseDHCP. You could edit any call of Install-ContainerHost from New-ContainerHost, but I do it another way.

Instead I edit Install-ContainerHost. One small change will do the trick. Not far from the top is where the parameters are set as script variables. Look for a line that reads:

$UseDHCP,

Modify this line so it reads:

$UseDHCP = $true,

Now every time I either run Install-ContainerHost or New-ContainerHost I’ll get the DHCP networking configuration instead of NATing.

So try this to create/configure a VM host, create a container, use Enter-PSSession to connect to the container, run IPConfig and … viola, you’ll have no DHCP address. Say what?

I was stumped. I tried it again. Nothing. I asked for help and by the time I got home, I got a tip from one of the folks in Redmond. It proved to be my “I’m a moron” moment of the day. If I’d thought about it, DHCP is all about broadcasts and MAC addresses. I have a single VLAN set up in the lab so broadcasts wasn’t the issue. What’s going on with MACs? A VM host has a MAC for itself. And then each container on the VM host that connects to the virtual switch has it’s own MAC address … but the network sees only one interface. Have you figured it out yet?

By default, Hyper-V has MAC spoofing disabled on every virtual NIC – a virtual NIC can only have 1 MAC address. What I needed to do was, at the host level, run the following to enable MAC spoofing on the VM host’s virtual NIC:

Get-VMNetworkAdapter -VMName containers3 | Set-vmNetworkAdapter -MacAddressSpoofing On

Now everything works

Technorati Tags: Windows Server 2016,Hyper-V,Virtualisation,Containers,Networking,PowerShell,Scripting