Month: July 2013

Comparing The Costs Of WS2012 Storage Spaces With FC/iSCSI SAN

Microsoft has released a report to help you “understand the cost and performance difference between SANs and a storage solution built using Windows Server 2012 and commodity hardware”.  In WS2012, they are referring to Storage Spaces and Scale-Out File Server.

From my own perspective, we’ve found the JBOD + Storage Spaces solution to be much cheaper than SAN storage, both on the upfront side (initial acquisition) and long term.  Adding disks and trays is cheaper – you get any manufacturer’s disk on the JBOD’s HCL rather than the 60% more expensive Dell/HP/etc disk from the same factory but with a “special” (lockdown) firmware.

ESG Lab tested the performance readiness and cost-effectiveness of Microsoft’s new storage solution and compared the results with two common storage solutions: an ISCSI and FC SAN. For performance testing, ESG Lab tested a tier-1 virtualized Microsoft SQL Server 2012 application workload and witnessed a negligible performance difference between all the tested storage configurations. In fact, when testing with as close to the exact same storage configuration as possible across each of the tested configurations, ESG Lab witnessed a slight performance benefit with Microsoft’s storage solution over iSCSI and FC SAN solutions.

ESG Lab also calculated what organizations could expect to spend when initially purchasing each storage configuration. The price difference was impressive. ESG Lab found that Microsoft’s storage solution can save organizations as much as 50% when compared with traditional iSCSI and FC SAN solutions. Another eye-opener for ESG Lab was around a features comparison between the storage configurations. With the upcoming release of Windows Server 2012 R2, Microsoft’s storage configuration is beginning to match traditional storage offerings feature for feature.

With similar performance, a matching feature set, less management complexity, and 50% cost-savings over a SAN, Microsoft’s Windows Server 2012 file server cluster with Storage Spaces over SMB 3.0 introduces a potentially disruptive storage solution to address any customer’s needs.

image

Yes, You Can Run A Hyper-V Cluster On Your JBOD Storage Spaces

What is the requirement of a cluster?  Shared storage.  What’s supported in WS2012/R2?

  • SAS SAN
  • iSCSI SAN
  • Fiber Channel SAN
  • FCoE
  • PCI RAID (like Dell VRTX)
  • Storage Spaces

What’s the difference between a cluster and a Hyper-V cluster?  You’ve enabled Hyper-V on the nodes.  Here are 2 nodes, each connected to a JBOD.  Storage Spaces is configured on the JBOD to create cluster shared volumes.  All that remains now is to enable Hyper-V on node 1 and node 2, and now you have a valid Hyper-V cluster that stores VMs on the CSVs

It’s completely supported, and a perfect Hyper-V cluster solution for the small/medium business, with the JBOD costing a fraction (search engine here and here) of the equivalent capacity SAN.

Stupid questions that you should not ask:

  • What file shares do I use to store my VMs on?  Where do you see “file shares” in the above text?  You store the VMs directly on the CSVs like in a Hyper-V cluster with a SAN, instead of storing file shares on the CSVs like in a SOFS cluster.
  • Can I run other roles on the hosts?  No.  You never should do that … and I include Exchange Server and SQL Server for the 2 people that I now hope have resigned from working in IT who asked that recently.
  • The required networks if you use 10 GbE are shown above.  Go look at converged networks for all possible designs; it’s the same clustered 2012/R2 Hyper-V networking as always.

Build WS2012 R2 Storage Pools, Virtual Disks, And CSVs Using PowerShell

I’ve been building, tearing down, and rebuilding Storage Spaces in the lab over and over, and that will continue for the next few years Smile Rather than spend a significant percentage of my life clicking on wizards, I decided to script what I want done.

The below script will:

  • Build a storage pool from all available disks
  • Prep 2 storage tiers from SSD and HDD
  • Create 3 different virtual disks with different configs (customize to your heat’s content!)
  • Then run the PrepCSV function to turn those virtual disks into CSVs just the way I like them

How do I like a CSV?  I like them formatted Smile and the names consistent all the way: virtual disk, cluster resource name, volume label, and CSV mount point in C:ClusterStorage.  None of that “Cluster Disk (X)” or “Volume 1” BS for me, thank you.

It might be possible to clean up the stuff in the function.  This is what I have working – it works and that’s the main thing.  There’s a lot of steps to get disk ID so I can create and format a volume, and then bring the disk back so I can turn it into a CSV.

What’s missing?  I have not added code for adding the SOFS role or adding/configuring shares.  I’m not at that point yet in the lab.

Function PrepCSV ($CSVName)
{
#Rename the disk resource in FCM
(Get-ClusterResource | where {$_.name -like “*$CSVName)”}).Name = $CSVName

#Get the disk ID
Stop-ClusterResource $CSVName
$DiskID = (Get-VirtualDisk -FriendlyName $CSVName).UniqueId
Start-ClusterResource $CSVName

#Format the disk
Suspend-ClusterResource $CSVName
Get-disk -UniqueId $DiskID | New-Partition -UseMaximumSize | Format-Volume -FileSystem NTFS -NewFileSystemLabel “$CSVName” -Confirm:$false
Resume-ClusterResource $CSVName

#Bring the CSV online
Add-ClusterSharedVolume -Name $CSVName
$OldCSVName = ((Get-ClusterSharedVolume $CSVName).SharedVOlumeInfo).FriendlyVolumeName
Rename-Item $OldCSVName -NewName “C:ClusterStorage$CSVName”
}

# The following Storage Pool and Virtual Disk cmdlets taken from Bryan Matthew’s TechEd  …
# … Session at http://channel9.msdn.com/Events/TechEd/Europe/2013/MDC-B217#fbid=TFEWNjeU9XP

# Find all eligible disks
$disks = Get-PhysicalDisk |? {$_.CanPool -eq $true}

# Create a new Storage Pool
New-StoragePool -StorageSubSystemFriendlyName “Clustered Storage Spaces on Demo-FSC1” -FriendlyName “Demo-FSC1 Pool1” -PhysicalDisks $disks

# Define the Pool Storage Tiers
$ssd_tier = New-StorageTier -StoragePoolFriendlyName “Demo-FSC1 Pool1” -FriendlyName SSD_Tier -MediaType SSD
$hdd_tier = New-StorageTier -StoragePoolFriendlyName “Demo-FSC1 Pool1” -FriendlyName HDD_Tier -MediaType HDD

 

#Transfer ownership of Available Storage to current node to enable disk formatting

Move-ClusterGroup “Available Storage” -Node $env:COMPUTERNAME

 

# Creation of a 200 GB tiered virtual disk with 5 GB cache
New-VirtualDisk -StoragePoolFriendlyName “Demo-FSC1 Pool1” -FriendlyName CSV1 –StorageTiers @($ssd_tier, $hdd_tier) -StorageTierSizes @(50GB,150GB) -ResiliencySettingName Mirror -WriteCacheSize 5GB
PrepCSV CSV1

 

# Creation of a 200 GB non-tiered virtual disk with no cache

New-VirtualDisk -StoragePoolFriendlyName “Demo-FSC1 Pool1” -FriendlyName CSV2 -Size 200GB -ResiliencySettingName Mirror -WriteCacheSize 0
PrepCSV CSV2

 

# Creation of a 50 GB virtual disk on SSD only with 5 GB cache

New-VirtualDisk -StoragePoolFriendlyName “Demo-FSC1 Pool1” -FriendlyName CSV3 –StorageTiers @($ssd_tier) -StorageTierSizes @(50GB) -ResiliencySettingName Mirror -WriteCacheSize 5GB
PrepCSV CSV3

 

EDIT1:

This script broke if the cluster group, Available Storage, was active on another node.  This prevented formatting, which in turn prevented adding the virtual disks as CSVs.  Easy fix: move the Available Storage cluster group to the current machine (a node in the cluster).

 

Getting Started With DataOn JBOD In WS2012 R2 Scale-Out File Server

Yesterday we took delivery of a DataOn DNS-1640D JBOD tray with 8 * 600 GB 10K disks and 2 * 400 GB dual channel SSDs.  This is going to be the heart of V2 of the lab at work, providing me with physical scalable and continuously available storage.

The JBOD

Below you can see the architecture of the setup.  Let’s start with the DataOn JBOD.  It has dual controllers and dual PSUs.  Each controller has some management ports for factory usage (not shown).  In a simple non-stacked solution such as below, you’ll use SAS ports 1 and 2 to connect your servers.  A SAS daisy chaining port is included to allow you to expand this JBOD to multiple trays.  Note that if scaling out the JBOD is on the cards then look at the much bigger models – this one takes 24 2.5” disks.

I don’t know why people still think that SOFS disks go into the servers – THEY GO INTO A SHARED JBOD!!!  Storage inside a server cannot be HA; there is no replication or striping of internal disks between servers.  In this case we have inserted 8 * 600 GB 10K HDDs (capacity at a budget) and 2 STEC 400 GB SSDs (speed).  This will allow us to implement WS2012 R2 Storage Spaces tiered storage and write-back cache.

image

The Servers

I’m recycling the 2 servers that I’ve been using as Hyper-V hosts for the last year and a half.  They’re HP DL360 servers.  Sadly, HP Proliants are stuck in the year 2009 and I can’t use them to demonstrate and teach new things like SR-IOV.  We’re getting in 2 Dell rack servers to take over the role as Hyper-V hosts and the HP servers will become our SOFS nodes.

Both servers had 2 * dual port 10 GbE cards, giving me 4 * 10 GbE ports.  One card was full height and the other modified to half height – occupying both ports in the servers.  We got LSI controllers to connect the 2 servers to the JBOD.  Each LSI adapter is full height and has 2 ports.  Thus we needed 4 SAS cables.  SOFS Node 1 connects to port 1 on each controller on the back of the JBOD, and SOFS Node 2 connects to port 2 on each controller.  The DataOn manual shows you how to attach further JBODs and cable the solution if you need more disk capacity in this SOFS module.

Note that I have added these features:

  • Multipath I/O: To provide MPIO for the SAS controllers.  There are rumblings of performance issues with this enabled.
  • Windows Standards-Based Storage Management: This provides us with with integration into the storage, e.g. SES

The Cluster

The network design is what I’ve talked about before.  The on-board 1 GbE NICs are teamed for management.  The servers now have a single dual port 10 GbE card.  These 10 GbE NICs ARE NOT TEAMED – I’ve put them on different subnets for SMB Multichannel (a cluster requirement).  That means they are simple traditional NICs, each with a different IP address.  I’ve used NetQOSPolicy to do QoS for those 2 networks on a per-protocol basis.  That means that SMB 3.0 and backup and cluster communications go across these two networks.

Hans Vredevoort (Hyper-V MVP colleague) went with a different approach: teaming the 10 GbE NICs and presenting team interfaces that are bound to different VLANs/subnets.  In WS2012, the dynamic teaming mode will use flowlets to truly aggregate data and spread even a single data stream across the team members (physical interfaces).

Storage Spaces

The storage pool is created in Failover Clustering.  While TechEd demos focused on PowerShell, you can create a tiered pool and tiered virtual disks in the GUI.  PowerShell is obviously the best approach for standardization and repetitive work (such as consulting).  I’ve fired up a single virtual disk so far with a nice chunk of SSD tiering and it’s performing pretty well.

image

First Impressions

I wanted to test quickly before the new Dell hosts come so Hyper-V is enabled on the SOFS cluster.  This is a valid deployment scenario, especially for a small/medium enterprise (SME).  What I have built is the equivalent (more actually) of a 2-node Hyper-V cluster with a SAS attached SAN … albeit with tiered storage … and that storage was less than half the cost of a SAN from Dell/HP.  In fact, the retail price of the HDDs is around 1/3 the list price of the HP equivalent.  There is no comparison.

I deployed a bunch of VMs with differential disks last night.  Nice and quick.  Then I pinned the parent VHD to the SSD tier and created a boot storm.  Once again, nice and quick.  Nothing scientific has been done and I haven’t done comparison tests yet.

But it was all simple to set up and way cheaper than traditional SAN.  You can’t beat that!

Deploy Roles Or Features To Lots Of Servers At Once

I’m deploying a large cluster at the moment and I wanted to install the Failover Clustering feature to all the machines without logging, doing stuff, logging out, and repeating.  This snippet of PowerShell took me 45 seconds to put together.  The feature is installing on 8 machines (Demo-FS1 to Demo-FS8) while I’m writing this blog post Smile

For ($i = 1; $i -lt 9; $i++)
{
    Install-WindowsFeature -ComputerName Demo-FS$i Failover-Clustering, RSAT-Clustering
}

The variable $i starts at 1, is used as part of the computer name that is remotely being updated, and then incremented in the next loop iteration.  The loop ends after the 8th iteration, i.e. the 8th server is updated.

Aint automation be-yoot-eeful?

Windows Firewall On Hyper-V Management OS (Host) Has Nothing To Do With Virtual Machines

Hyper-V veterans should respond with “of course it doesn’t”.  But fairly often I encounter a question online where someone is trying to disable Windows Firewall for a VM … and they start mucking around with Windows Firewall in the Management OS (the host OS). 

Hyper-V, as the name suggests, is a hypervisor.  It’s a type 1 hypervisor, similar to but different to vSphere.  The Host OS networking has nothing to do with VM networking.  The host has it’s own network interface.  The VMs have their own network interfaces.  They might share the same physical NIC team or NIC but their commonalities end there.  The host and it’s VMs have independent virtual NICs, MAC addresses and IP addresses, each operating independently from each other, even if they share the same physical NIC or NIC team.

If you configure the Windows Firewall in the Management OS then you are only affecting the host.  To affect a VM then configure the Windows Firewall in the VM’s guest OS.

The independence follows through to the physical network.  Because the host and VMs are independent, then rules on the physical network firewall apply to the address that you are including in the rule, i.e. a rule affecting the host IP address has nothing to do with the VMs.

BTW, if you are looking to do VM firewalling at the host layer then look for a virtual switch extension, such as the one from 5nine.

Update Rollup 3 For System Center 2012 Service Pack 1

Microsoft released UR3 for System Center 2012 SP1 overnight.  It contains bug fixes for:

  • App Controller
  • Operations Manager
  • Virtual Machine Manager

Download links and installation instructions are on the site.

My advice: considering the quality of patches coming out of Redmond recently, I’d wait a month before installing these updates.

EDIT1:

FYI, it appears some download links aren’t active just yet.

Constraining SMB Powered Redirected IO Traffic To Selected Networks In A WS2012 Hyper-V Cluster

I was recently doing some System Center work on a customer site when they experienced a peculiar problem with their Windows Server 2012 (WS2012) Hyper-V cluster.  They were recovering a very large virtual machine.  OK – nothing weird there.  What was weird was that they lost management access to the hosts as soon as the restore job started.  VMs were fine – it’s just that they couldn’t remotely manage their hosts any more.

I was told of the problem the following morning.  It wasn’t in my project scope but I decided to dig around; I immediately suspected Redirected IO was to blame and that’s a topic I’ve written a word or two about in the past.

Recap: What is Redirected IO?

Redirected IO is a CSV feature where non-owners of the CSV will redirect their storage reads/writes to the storage via the owner of the CSV.  In other words:

  • The owner temporarily has exclusive read/write access to the CSV
  • All other nodes in the cluster read/write from/to the CSV via the owner (or CSV coordinator)

Why is this done?  There are two reasons in WS2012:

  • There is a metadata operation taking place: this could be a VM start (very very quick), a dynamic virtual hard disk expansion, or even the creation of a fixed virtual hard disk.
  • A node loses direct connectivity to the storage and uses redirect IO to avoid VM storage corruption or VM outage.

Note that redirected IO was used for Volume Shadow Copy Snapshot (VSS powered backup) operations in W2008 R2 but that is no longer the case in WS2012; it uses a new distributed snapshot mechanism to simply the backup.

Why Did Redirected IO Kick In Here?

Restoring the VM was a backup operation, right?  No; it’s the complete opposite – it’s a restore operation and has absolutely nothing to do with VSS.  Effectively the restore is creating files on a CSV … and that is one big mutha of a metadata operation.  I wasn’t there at the time and I don’t know the backup tool, but I suspected that the admin restored the VM to a host that was not the owner of the target CSV.  And that created tonnes of redirected IO:

  1. The backup application restored some 500 GB of VM to the destination node
  2. The node was not the CSV owner so redirected IO kicked in
  3. 500 GB of data went redirecting from the “restore” host to the CSV owner across 1 GbE networking and then into the storage
  4. Redirected IO finished up once the metadata operation (the restore) was completed

That’s my Dr. House theory anyway.  I think I have it right – all the pieces fit.  Of course, sometimes the patient died in House.

Didn’t They Control The Redirected IO Network?

They thought they did.  The actually implemented the correct solution … for W2008 R2.  That solution manipulates (if necessary) the network metric value to make the CSV network the one with the lowest metric.  This is a WS2012 R2 cluster and things are different.

The clue why things are different is in the above illustration.  There are two layers of redirected IO (the new lower block approach is 4 times faster) and bother are powered by SMB 3.0.  What’s one of the new features of SMB 3.0?  SMB Multichannel.  And what does SMB Multichannel do?  It finds every connection between the SMB client (the non-CSV owner) and the SMB server (the CSV owner) and uses all those networks to stream data as quickly as possible.

That’s great for getting the SMB transfer done as quickly as possible, but it can be damaging (as I suspected it was here) if left unmanaged.  And that’s why we have features in WS2012 like QoS and SMB Constraints.

The Problem & Solution

In this case the hosts had a number of networks.  The two that are relevant are the cluster network and the management network:

  • Cluster network: A private network with 2 * 1 GbE NICs in a team
  • Management network: 2 * 1 GbE in a team

Two networks and both were eligible for SMB 3.0 to use for SMB Multichannel during Redirected IO.  And that’s what I suspect happened … 500 GB of data went streaming across both NIC teams for the duration of the restore operation.  That flooded the management network.  That made it impossible for administrators to remotely manage or RDP into the hosts in the cluster.

The solution in this case would be to use SMB constraints to control which networks would be used by SMB Multichannel at the client end.  This cluster is simple enough in that regard.  It uses iSCSI storage and SMB Multichannel should only be used by the team … so the team’s interfaces would be listed in the cmdlet.  Now SMB Multichannel will only use the team.

I didn’t implement the solution – that is outside my scope on the project – but I’ve made the recommendation.  I’ll update if I get any news.

In these types of clusters I’d recommend this solution as a part of the build process.  It gets a bit more complicated if you’re using SMB 3.0 storage: just apply constraints to select any NIC that has a valid SMB Multichannel role, e.g. storage and redirected IO.

A Very Important Article About Health Of Virtual DCs On Hyper-V

I strongly urge you (in other words, do it or else) to head over to Hyper-V.nu to read an article that Hans Vredevoort (Virtual Machine MVP) wrote on the July 2013 Update Rollup and what it does to prevent total corruption of Active Directory domain controllers that are running as virtual machines on Hyper-V.

Nuff said … go read it.

KB2872325 – VMs On WS2012 Hyper-V May Not Be Able To Create Or Join Guest Cluster

A new KB article from Microsoft appeared today for when guest cluster nodes in Windows Server 2012 Hyper-V may not be able to create or join a cluster.

Symptoms

If using the “Create Cluster Wizard” the cluster may fail to create. Additionally, the report from the wizard may have the following message:

An error occurred while creating the cluster.
An error occurred creating cluster ‘<clustername>’.
This Operation returned because the timeout period expired

Note: The above errors can also be seen anytime that communications between the servers that are specified to be part of the cluster creation do not complete. A known cause is described in this article.
In some scenarios, the cluster nodes are successfully created and joined if the VMs are hosted on the same node, but once the VMs are moved to different nodes the communications between the nodes of the guest cluster starts to fail. Therefore the nodes of the cluster may be removed from the cluster.

Cause

This can occur due to packets not reaching to the virtual machines when the VMs are hosted on Windows Server 2012 failover cluster nodes, due to a failover cluster component that is bound to the network adapters of the hosts.  The component is called the “Microsoft Failover Cluster Virtual Adapter Performance Filter” and it was first introduced in Windows Server 2012.

The problem only effects the network packets addressed to cluster nodes hosted in virtual machines.

Workaround

It is recommended that if the Windows Sever 2012 failover cluster is going to host virtual machines that are part of guest clusters, you should unbind the “Microsoft Failover Cluster Virtual Adapter Performance Filter” object from all of the virtual switch network adapters on the Windows Server 2012 Failover Cluster nodes.

Instructions for the workaround (GUI and PowerShell options) are in the KB article.