Aidan Finn, IT Pro

Hold Onto Your Knickers, I’m About To Compliment Windows Phone On Something!

As a listener to TWiT’s Windows Weekly, I’ve heard Paul Thurrott and Leo Laporte talk about Xbox Music Pass. Also called Xbox Music, it is a leasing agreement, where for a few dollars a month, you can stream, download/play (on up to 5 devices) unlimited music.

Because of how Microsoft has typically put arbitrary regional restrictions on apps and contents over the years, I assumed (Arrgh! and regulars know how I hate assumptions) that a sweet deal like this would not be available outside of the USA, or maybe the usual 7 countries. This morning, one of my colleagues came over and asked if I’d tried Xbox Music because he was loving it. I was … surprised.

I just checked Andrew Birch’s amazing feature availability matrix on Andrew Tech Help, and lo and behold, XBox Music is available in more (not all, not even most) countries than I expected.

My colleague has Xbox Music set up on his Windows Phone. It’s nice and seamless. The leasing agreement (you keep the music as long as your subscription is active) allows you to download to and play music on up to 5 devices (just like with Windows Store Apps). We went over to his PC so I could explore. OK, I expect to find mainstream stuff like Radiohead, David Bowie, or X-Craptor, but what about the stuff I listen to? It’s the stuff you don’t find on music shelves (actually there are none of those anymore), but would Microsoft have bothered to add my quirky music in addition to the usual Crappy Lee Jepson or James Farter? Yup, the albums were there. I’m impressed.

So, in Ireland, for €99 per year, you can have unlimited music listening, legally. New stuff and old (the stuff I searched for was released in late 80’s and mid 90’s). You can listen on your Windows Phone or via the (XBox) Music app in Windows 8. Searching is a fantastic experience in Windows 8 (for everything: settings, files, app content) so finding the music you want to listen to or try is superb.

There is a free 30 day trial. I’m told it restricts the number of hours you can listen until you start to pay. It looks like, if you start the trial it will automatically convert into a paid subscription unless you cancel.

Albums seem to be a thing of the past for most people. You can try a new artist or style of music with no financial commitment (better than laying down €22 for a CD in a “store”), pick and choose songs you like, download them to your Windows Phone to listen to on a plane (with no Internet connection), and create playlists. And it’s all legal. And €100 is a paltry annual amount.

Right now Xbox Music is available on Windows Phone 8, Xbox 360 (stream only and requires Xbox Live Gold), Windows 8 and Windows RT. Windows 7 and Windows Phone 7 are not included, and XP is well out of mainstream support so it’ll never be included. Other services such as Spotify have great cross-platform support. They’ll have that advantage while Xbox Music doesn’t support Android and iOS. Allegedly, Microsoft will bring support to those two mobile device OSs sometime this year. Back in November, Brad Chacos on PCworld.com reported:

Android and iOS support will come "within 12 months

It’s good to see some of the “3 screens and a cloud” stuff appearing in the consumer space on Windows 8/RT, Xbox, and Windows Phone 8.

The People hub in WP8 is superb if the phone is your personal device (I still hate that it opts in social network contacts lists by default on my work device), and live tiles are better than dead icons. Those are innovations by Microsoft (sure, MSFT are being sued over Live Tiles so there is some question there [covering my a$$]) that we should be thankful for, and that might be contributing to the aging of iOS.

Handset hardware quality is infinitely better than it was in the last generation. The few of us in the office that have the HTC 8x love the hardware.

I still cannot forgive the arbitrary regional fencing of features. There is no licensing issue for podcasts. They are put out on the net via RSS feeds and shared via loads of catalogs. If Microsoft can share podcasts via the Marketplace in some countries then there is absolutely no reason not to do it in all countries. That sort of bollo% is what led me to assume (arrgh!) that Xbox Music (actual licensed content) wouldn’t be available here.

Technorati Tags: Windows 8

The Big Changes In WS2012 Cluster Shared Volume (CSV)

Microsoft made lots of changes with CSV 2.0 in Windows Server 2012. But it seems like that message has not gotten through to people. I’ve responded to quite a few comments here on the blog and I’m seeing stuff on forums. What’s really annoying is that when you tell people that X has changed, they don’t listen.

I would strongly recommend that people take some time (I don’t care about excuses) to watch the TechEd presentation, Cluster Shared Volumes Reborn in Windows Server 2012: Deep Dive, by Rob Hindman and Amitabh Tamhane (Microsoft). There are lots of changes. But I want to focus on the big ones that people repeatedly question.

OK, what are the major changes?

There IS NO Redirected IO in WS2012 CSV Backup

Let me restate that in another way: Windows Server 2012 does not use Redirected IO to backup CSVs.

This has been made possible thanks to substantial changes in how VSS places VMs that are stored on CSV into a quiescent state. The backup agent (VSS Requestor) kicks off a backup request with a list of virtual machines. The Hyper-V Writer identifies the storage location(s) of the VMs’ files. A new component, the CSV Writer, is responsible for coordinating the Hyper-V nodes in the cluster … meaning all VMs on a CSV that is being backed up to be placed into a quiescent state at the same time. This allows for a single distributed VSS snapshot of each CSV. That allows the provider (hardware, software or system) to go to work and get the snapshot.

This is much simpler than what CSV did in Windows Server 2008 R2. [The following does not happen in WS2012] There was no CSV Writer. There was no coordination, so Redirected IO was required. The node performing a snapshot needed exclusive access to the volume so all IO went through it for the time being. A lot of people knew that bit up to there. The bit that most people didn’t know was that each node (hosting VMs that were being backed up) took snapshots of each CSV that was being backed up. And that could cause problems.

I’ve heard several times now from people who’ve experienced issues with volumes going offline during backup. There were two causes that I’ve seen, and both were related to a third party hardware VSS provider:

Using a hardware VSS provider that did not support CSV
The rapidly rotating and repeated snapshot process caused chaos in the SAN with the hardware snapshots

But, all that is G-O-N-E when backing up CSV on Windows Server 2012:

There is no redirected IO
There is a single VSS snapshot performed

SCSI3 Reservation Starvation Should Go Away

Every node in a Hyper-V cluster used SCSI3 persistent reservations and SCSI3 reservations to connected to CSVs. Every SAN has a finite number of those persistent reservations and reservations. The SCSI3 persistent reservations was a bottleneck. No manufacturer shares that number, and it’s a hell of a lot smaller than you’d expect – we typically find out about it during a support call. To compound this, each host required a number of SCSI3 persistent reservations, and that multiplied based on:

Number of hosts in the cluster
Number of CSVs
Number of storage channels per host (possibly even a multiple of the number of physical HBAs/NICs, depending on the SAN)

What happens when you deploy too many nodes, CSVs, or storage channels? CSVs go offline. Yup. The SAN is starved of resources to connect the hosts to the LUNs. I saw this with small deployments with an entry level SAN, 3 hosts, and 5 CSVs. And it aint pretty.

Imagine a cluster with 64 nodes!?!?! With Windows Server 2012, each node gets a static key instead of using the legacy persistent reservation multiplication. That means your SAN can support more CSVs and more hosts running Windows Server 2012 than it would have with Windows Server 2008 R2. Note that the static key is assigned when the node is added to the cluster.

You can find the static keys in the registry of your cluster nodes in HKEY_LOCAL_MACHINEClusterNodes<Node Number>ReserveID (REG_QWORD). You can identify which node number is which host by the NodeName (REG_SZ) value. You can see an example of this below.

This new system, which replaces persistent reservations, gives you better cluster infrastructure scalability, but it doesn’t eliminate the scalability limits of your SAN.

Technorati Tags: Failover Clustering,Windows Server 2012,Hyper-V,Virtualisation,Storage,Backup

KB2779768–Experiencing Issues When Using NLB In VMs On WS2012 NIC Teaming

The cumulative update, KB2779768 (released through Windows Update), fixes stability issues when you use Windows Server Network Load Balancing in virtual machines, that connect to the network using a Windows Server 2012 NIC Team. This issue range from disconnections to sad faces of death (aka blue screen of death [BSOD]).

The observant of you will notice there is no documentation on the referenced page, and the referenced KB articles do not exist, at least publicly. However, it has been confirmed that this update does include the fix for the above issue.

Technorati Tags: Hyper-V,Virtualisation,Networking,Windows Server 2012

Memory Leak Issues Being Reported With KB2799728

I mentioned a new KB article/hotfix yesterday that resolves an issue with backup of CSV on WS2012 Hyper-V clustered nodes.

Tim Boothby and Rich Lilly commented on the post that people are reporting memory leak issues after installing this hotfix. My advice: don’t install the update unless:

You really need it OR
It appears that the issue is resolved

Technorati Tags: Failover Clustering,Backup,Hyper-V,Virtualisation

System Center Hyper-V Monitoring Pack For Windows Server 2012

You can find the documentation on TechNet for the System Center Operations Manager management pack for monitoring WS2012 Hyper-V. This management pack supports Operations Manager 2012.

The management pack will monitor:

The host
Virtual machines
RemoteFX
Hyper-V Replica

You can download this System Center management pack from here.

Technorati Tags: Windows Server 2012,Hyper-V,Virtualisation,System Center

Windows Server 2012 NIC Teaming Part 6 – Support Policies

Windows Server 2012 NIC Teaming Part 1 – Back To Basics

Windows Server 2012 NIC Teaming Part 2 – What’s What?

Windows Server 2012 NIC Teaming Part 3 – Switch Connection Modes

Windows Server 2012 NIC Teaming Part 4 – Load Distribution

Windows Server 2012 NIC Teaming Part 5 – Configuration Matrix

Windows Server 2012 NIC Teaming Part 6 – NIC Teaming In The Virtual Machine

This post is going to focus on support policies. I expect most mistakes in NIC teaming will be where people try to do things that are not supported. I want to summarise the support policies here. This content, once again, is taken from Microsoft’s document on NIC teaming.

Feature Support

There are lots of networking features in Windows Server 2012. Some support NIC teaming, some work great with NIC teaming, some (like SR-IOV) ignore NIC teaming in the host but are fine with NIC teaming in the guest OS, and some flat out do not support NIC teaming.

Feature	Comments
Datacenter Bridging (DCB)	Works below NIC Teaming in the NIC so is supported if the team members support it.
IPsec Task Offload (IPsecTO)	Supported if all team members support it.
Large Send Offload (LSO)	Supported if all team members support it.
Receive side coalescing (RSC)	Supported in hosts if any of the team members support it. Not supported through Hyper-V switches.
Receive side scaling (RSS)	NIC teaming supports RSS in the host. The Windows Server 2012 TCP/IP stack programs the RSS information directly to the Team members.
Receive-side Checksum offloads (IPv4, IPv6, TCP)	Supported if any of the team members support it.
Remote Direct Memory Access (RDMA)	NIC team members cannot support RDMA because the protocol bypasses the networking stack.
Single root I/O virtualization (SR-IOV)	You cannot do NIC teaming with SR-IOV enabled NICs. Do the teaming in the guest OS with 2 Hyper-V external switches.
TCP Chimney Offload	Not supported through a Windows Server 2012 team.
Transmit-side Checksum offloads (IPv4, IPv6, TCP)	Supported if all team members support it.
Virtual Machine Queues (VMQ)	Supported when teaming is installed under the Hyper-V switch.
QoS in host/native OSs	Supported, but use of minimum bandwidth policies will degrade throughput through a team.
Virtual Machine QoS (VM-QoS)	VM-QoS is affected by the load distribution algorithm used by NIC Teaming. For best results use HyperVPorts load distribution mode.
802.1X authentication	Not compatible with many switches. Should not be used with NIC Teaming.

Team Members

Between 1 and 32 team members (2 in a guest OS NIC team) on the Windows Server 2012 hardware compatibility list (WHQL or logo’d). The NICs can be of mixed model and manufacturer.

The active team members can be of different speed, but must be operating at the same speed. A hot-spare NIC (in a 2-team-member team) maybe run at a different speed than the active team members.

You cannot use NICs other than Ethernet in your team – no Wi-Fi, Bluetooth, etc.

Guest OS NIC Teams

See my previous post on Guest OS NIC teaming.

The preferred (that is, not the only way, just the preferred way) to support multiple VLANs in a single VM is to:

Create multiple vNICs in the VM, one per required VLAN
Enable trunking on the physical switch(es)
Configure a VLAN ID for each virtual NIC, one per required VLAN
Rename the virtual NICs (the VLAN ID as part of the name) using Rename-NetAdapter

Teaming

You cannot create a team made up of other teams. You cannot use the team interface of another NIC teaming technology as a WS2012 team member or vice versa.

Microsoft says:

… it is STRONGLY RECOMMENDED that no system administrator ever run two teaming solutions at the same time on the same server.

In other words, don’t use HP/Dell/Intel/Broadcom NIC teaming on a machine where you intend to use WS2012 NIC teaming. And remember – Microsoft does not support any 3rd party NIC teaming and never has.

You cannot create a NIC team that is made up of management OS NIC teams. MPIO is not NIC teaming and we can use MPIO for multiple management OS virtual NICs for iSCSI.

MAC Address and Switch Independent/Address Hash Teams

This is an odd one. Remember that this type of team receives inbound traffic on a single IP team member? That’s because it uses a single MAC address to register the IP address. Removing the primary team member from the team, and reusing it for something else on the network can cause a MAC address conflict with unpredictable results. To prevent/resolve this, disable and re-enable the team interface – that causes the team to pick a new MAC address to register on the network.

VLANs

To support VLANs on the NIC team or in guests:

Set the team members physical switch ports to trunk mode
Do not filter VLANs on the team members

Virtual Switch

When using a NIC team for a virtual switch, there should be just a single team interface. This team interface is used just for the virtual switch. There should be no VLAN filtering in the team or the team interface. All VLAN filtering should be done by the switch (properties of virtual NIC), or within the guest OS.

—

This is the last planned post in this series, but I might revisit with more if I think of something. I am not doing any technical posts on how to create/configure/use teams – you can read all about that in Windows Server 2012 Hyper-V Installation and Configuration Guide (available on pre-order on Amazon, and due in Feb/March) where you’ll find in-depth discussion of NIC teaming, as well as all the other pieces of networking that will make your WS2012 Hyper-V hosts sing. The chapter I wrote on Hyper-V networking is a monster, taking you from the very basics, through virtual switches, extensibility, VLANs, NIC teams, hardware offloads/optimisations, QoS, and converged fabrics … with lots of examples and PowerShell:

And no, there are no preview/beta copies of the book.

Technorati Tags: Windows Server 2012,Hyper-V,Networking

Dell Releases vOPS Server Explorer 6.3 For Hyper-V and VMware – And There Is A Free Version

Last week, I (and a few others) was lucky to be offered a demonstration of vOPS Server Explorer 6.3, a product of vKernel (now a part of Dell). This product, launched today, adds to the management functionality of System Center or vCenter. vOPS Server Explorer also supports Red Hat Enterprise Virtualization.

What I saw was a very interesting intelligence system. It gathered data from your management system, combined it, analysed it, and gave you a better view of how your infrastructure is operating, resources, are being used, and virtual machines are performing. You’ll also find very nice features such as Zombie VMs (to control VM sprawl – maybe relocate them to the VMM Library and delete at a later point), and Rightsizing Savings (over allocating resources to VMs is costly in resources and can actually reduce overall host performance).

There is a free version which will alert you to issues. If you “upgrade” it to a trial of the paid-for version then you can use vOPS Server Explorer to fix the issues. You get a time-limited trial, which you can pay for to complete the upgrade or downgrade again to the free product.

One of the features which I think larger or change-controlled organisations will like was the change log (Change Explorer). Configuration changes are tracked and associated with the person responsible. And there’s an undo option!

You can learn more about the release here and download it here. The release information site includes a bunch of youtube videos with demonstrations of vOPS Server Explorer in action. The installation and configuration guide is here.

Installation is easy: vOPS Server Explorer is delivered as a virtual machine, and the guide will walk you through getting it running.

Note that the Hyper-V requirements are:

Systems Center Operations Manager 2007 R2 and Systems Center Virtual Machine Manager 2008 R2
Or
Systems Center Operations Manager 2012 and Systems Center Virtual Machine Manager
2012

Java 1.6 or higher is required.

Technorati Tags: Hyper-V,Virtualisation,VMware

KB2799728–VM Enters Paused State Or CSV Goes Offline When Backup WS2012 Hyper-V Cluster

Please note the below comments. There are memory leak issues being reported with this hotfix.

Please pay special attention to this hotfix. It’s the sort of one I expect to see on forums and be asked about for the next 18 months. I recommend making this patch a standard part of your install of WS2012 Hyper-V clusters.

The scenario is when a virtual machine enters a paused state or a CSV volume goes offline when you try to create a backup of the virtual machine on a Windows Server 2012-based failover cluster.

Consider the following scenario:

You enable the Cluster Shared Volumes (CSV) feature on a Windows Server 2012-based failover cluster.

You create a virtual machine on a CSV volume on a cluster node.

You start the virtual machine.

You try to create a backup of the virtual machine on the CSV volume by using Microsoft System Center Data Protection Manager (DPM) or any backup software that uses the Microsoft Software Shadow Copy Provider.

In this scenario, one of the following issues occurs:

The backup is created, and the virtual machine enters a paused state.

The CSV volume goes offline. Therefore, the virtual machine goes offline, and the backup is not created.

Additionally, the following events are logged in the Cluster log and System log respectively:

Software snapshot creation on Cluster Shared Volume(s) (‘volume location‘) with snapshot set id ‘snapshot id‘ failed with error ‘HrError(0x80042308)(2147754760)’. Please check the state of the CSV resources and the system events of the resource owner nodes.

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: Date and time
Event ID: 5120
Task Category: Cluster Shared Volume
Level: Error
Keywords:
User: SYSTEM
Computer: Computer name
Description: Cluster Shared Volume ‘Volume1’ (‘name’) is no longer available on this node because of ‘STATUS_IO_TIMEOUT(c00000b5)’. All I/O will temporarily be queued until a path to the volume is reestablished.

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          Date and time
Event ID:      5142
Task Category: Cluster Shared Volume
Level:         Error
Keywords:
User:          SYSTEM
Computer:      Computer name
Description: Cluster Shared Volume ‘Volume3’ (‘Cluster Disk 4’) is no longer accessible from this cluster node because of error ‘ERROR_TIMEOUT(1460)’. Please troubleshoot this node’s connectivity to the storage device and network connectivity.

The virtual machine enters a paused state because the Ntfs.sys driver incorrectly reports the available space on the CSV volume when the backup software tries to create a snapshot of the CSV volume. Additionally, the CSV volume goes offline because the CSV volume does not resume from a paused state after an I/O delay issue or an I/O error occurs.
Note The CSV volume is resilient.

A supported hotfix is available from Microsoft.

There is more:

After you install the hotfix, CSV volumes do not enter paused states as frequently. Additionally, a cluster’s ability to recover from expected paused states that occur when a CSV failover does not occur is improved.

To avoid CSV failovers, you may have to make additional changes to the computer after you install the hotfix. For example, you may be experiencing the issue described in this article because of the lack of hardware support for Offloaded Data Transfer (ODX). This causes delays when the operating system queries for the hardware support during I/O requests.

In this situation, disable ODX by changing the FilterSupportedFeaturesMode value for the storage device that does not support ODX to 1. For more information about how to disable ODX, go to the Microsoft website.

Technorati Tags: Failover Clustering,Hyper-V,Virtualisation,System Center,Backup