Category: Windows Server

How Much RAM & CPU Does Window Server Deduplication Optimization Require?

I’ve been asked about resource requirements for the dedupe optimization job before but I did not have the answer before now.

Processor

The CPU side is … not clear.  The dedupe subsystem will schedule one single-threaded job per volume. That means a machine with 8 logical processors is only 1/8th utilized if there is a single data volume. Microsoft says:

To achieve optimal throughput, consider configuring multiple deduplication volumes, up to the number of CPU cores on the file server.

That seems pretty dumb to me. “Go ahead and complicate volume management to optimize the dedupe processing”. Uhhhhh, no thanks.

Memory

Microsoft tells us that 1-2 GB RAM is used per 1 TB of data per volume.  They clarify this with an example:

Volume Volume size Memory used
Volume 1 1 TB 1-2 GB
Volume 2 1 TB 1-2 GB
Volume 3 2 TB 2-4 GB
Total for all volumes 1+1+2 * 1GB up to 2GB 4 – 8 GB RAM

By default a server will limit the RAM used by the optimization job to 50% of total RAM in the server.  So if the above server had just 4 GB RAM, then only 2 GB would be available for the optimization job.  You can manually override this:

Start-Dedupjob <volume> -Type Optmization  -Memory <50 to 80>

There is an additional note from Microsoft:

Machines where very large amount of data change between optimization job is expected may require even up to 3 GB of RAM per 1 TB of diskspace.

So you might see RAM become a bottleneck or increase pressure (in a VM with Dynamic Memory) if the optimization job hasn’t run in a while or if lots of data is dumped into a deduped volume.  Example: you have deployed lots of new personal (dedicated) VMs for new users on a deduped volume.

How Many SSDs Do I Need For Tiered Storage Spaces?

This is a good question.  The guidance I had been given was between 4-8 SSDs per JBOD tray.  I’ve just found guidance that is a bit more precise.  This is what Microsoft says:

When purchasing storage for a tiered deployment, we recommend the following number of SSDs in a completely full disk enclosure of different bay capacities in order to achieve optimal performance for a diverse set of workloads:

Disk enclosure slot count Simple space 2-way mirror space 3-way mirror space
12 bay 2 4 6
24 bay 2 4 6
60 bay 4 8 12
70 bay 4 8 12

Minimum number of SSDs Recommended for Different Resiliency Settings

KB2779069 – Hotfix To Determine Which Cluster Node Is Blocking GUM Updates In WS2008R2 & WS2012

First some background …

A cluster is made up of (normally) 2 or more servers.  They use a distributed database to keep a synchronised copy of the configuration of the HA resources, e.g. HA VMs on a Hyper-V cluster.  Something called the Global Update Manager (GUM) is used to coordinate consistent updates to resource configurations across the cluster nodes. 

When a node contains an update that has to be shared with other nodes an initiator node first obtains a GUM lock. Then, the node shares the update by using a Multicast Request Reply (MRR) message to the other nodes. After this update is sent, the initiator node waits for a response from other nodes before you continue. However, in certain conditions, one of the nodes does not reply to the GUM request in time because the node is stuck for some reason. Currently, there is no mechanism to determine which node is stuck and does not reply to the GUM request.

That’s just changed thanks to a hotfix that is now available that adds two new cluster control codes to help you determine which cluster node is blocking a GUM update in Windows Server 2008 R2 and Windows Server 2012.

After you install this hotfix, two new cluster control codes are added to help the administrator resolve the problem. One of the cluster control codes returns the GUM lock owner, and the other control code returns the nodes that are stuck. Therefore, administrator can restart the stuck nodes to resolve the problem. For more information about the new control codes, go to the following Microsoft:

Notes

  • The Cluster service has a facility that is called GUM. GUM is used to distribute a global state throughout the cluster.
  • Only one cluster node can send GUM messages at any time. This node is called the GUM lock owner.
  • The GUM lock owner sends an MRR message to a subset of cluster nodes, and then waits for the nodes send message receipt confirmations.
  • Run some iterations of these control codes to confirm that the node is stuck.
  • After the CLUSCTL_CLUSTER_GET_GUM_LOCK_OWNER control code is called, you have to close the cluster handle. Then, you reopen the cluster handle by using the GUM lock owner node name that is returned by the control code. If you do not perform this action, the CLUSCTL_NODE_GET_STUCK_NODES control code may return an incorrect result.

You can get this hotfix from here.

KB2905412 – Stop Error 0xD1 On Windows-Based Computer With Multiple Processors

Not strictly a Hyper-V issue, but you’ll understand why I am blogging about this one; A hotfix has been released for when a stop error 0xD1 on a Windows-based computer with multiple processors.

Symptoms

Your multiprocessor Windows-based computer crashes every two to three days. Additionally, Stop error 0xD1 is generated when the computer crashes.

Cause

This problem occurs because of a race condition that exists in the TCP/IP driver in a multiprocessor environment. If duplicate TCP segments are received on different processors, they may be sequenced incorrectly, and this triggers the crash.

A hotfix is available. 

KB2898774 – Data Loss Occurs On SCSI Disk That Turns Off In WS2012-Based Failover Cluster

Microsoft has released a KB article to avoid data loss occurring when a SCSI disk turns off in a Windows Server 2012-based failover cluster.

Symptoms

Consider the following scenario:

  • You deploy a Windows Server 2012-based failover cluster. The cluster contains two nodes (node A and node B).
  • A SCSI disk is used for the failover cluster. The disk is a shared disk and is accessible by both node A and node B.
  • Node A restarts or crashes. Then, the cluster fails over to node B.
  • Node A comes back online.
  • Node B is shut down and the cluster fails over to node A.
  • You write some data to the disk.
  • The disk turns off unexpectedly. For example, the device losses power.

In this scenario, the data that you write to the disk is lost.
Notes

  • This issue also occurs when the cluster contains more than two nodes.
  • This issue does not occur if the SCSI disk supports the SCSI Primary Commands – 4 (SPC-4) standard.

To resolve this issue, install update rollup 2903938.  It’s an update rollup, so update rollup rules apply – either test like nuts in a lab or wait a month before you approve/deploy it.

Remote Server Administration Toolkit For Windows 8.1

Microsoft has released the RSAT for Windows 8.1.  This is the toolkit you will install on administrators’ Windows 8.1 PC to manage Windows Server 2012 R2 (WS2012 R2) and older.

Remote Server Administration Tools for Windows 8.1 Preview includes Server Manager, Microsoft Management Console (MMC) snap-ins, consoles, Windows PowerShell cmdlets and providers, and command-line tools for managing roles and features that run on Windows Server 2012 and Windows Server 2012 R2 Preview. In limited cases, the tools can be used to manage roles and features that are running on Windows Server 2008 R2 or Windows Server 2008. Some of the tools work for managing roles and features on Windows Server 2003.

EDIT: The link for this package from the below support matrix still points to the RSAT Preview.  I guess it will get updated soon. The link was updated (reused) to the GA bits.

There are a few important support notes:

  • The IP Address Management (IPAM) console should not be used (not supported) to manage WS2012.
  • The WS2012 R2 and Windows 8.1 PowerShell cmdlets should not be used to manage WS2012.  It is not blocked, but it is not supported.  It is supported to manage WS2012 Hyper-V using the Hyper-V GUI tools (FCM and Hyper-V Manager) from RSAT for Windows 8.1.
  • To manage WS2012 iSCSI target using PowerShell on Windows 8.1 then you must import the WS2012 (RSAT for Windows 8) PowerShell module.

Yes, I agree; This is very messy.

A support matrix for RSAT for Windows 8 and RSAT for Windows 8.1 has also been posted.

You’ll not that some tools are not in RSAT for Windows 8.1, and some are specifically listed as deprecated:

  • SMTP Server Tools: Not included.
  • Storage Explorer Tools: Not included.
  • Storage Manager for Storage Area Network (SAN) Tools: No included.
  • Windows System Resource Manager Tools: Deprecated and not in WS2012 R2.

Using Microsoft Deployment Toolkit To Deploy Servers

I’ve started using MDT 2013 Preview to deploy the physical servers and template VMs in the lab at work.  I’ve never really used this free imaging + task sequence solution for anything other than desktop OSs so this was a first for me.

Note: In a larger environment, you should use System Center 2012 R2 to deploy your servers.  For example VMM 2012 R2 now deploys hosts and Scale-Out File Server clusters (including the nodes). 

Using MDT means I can push out customised deployments very quickly.  I have a single WS2012 R2 Datacenter image that I will update with patches from time to time.  I have imported drivers, e.g. Chelsio and LSI, into MDT  to plug and play during deployment.  And I have multiple task sequences for different types of machine.  Each task sequence is configured to add roles/features:

  • Plain one for a preparing a template VM
  • Domain controller with DNS and DHCP
  • File server with failover clustering, dedup, VSS support, DCB, MPIO, etc, for use as a SOFS node – I actually have one for physical and one for virtual
  • A Hyper-V host with Hyper-V, failover clustering, DCB, MPIO, etc

I use WDS for the boot image, so I can run MDT bare metal deployments over the network, initiated by PXE boots.

Over time I’ll probably add some scripts to the MDT deployment share that will enable further customisations during the task sequences, such as configuring NIC teams, setting Jumbo Frames, etc.

If you want to learn more about MDT then I have to recommend Deployment Fundamentals, Vol. 4: Deploying Windows 8 and Office 2013 Using MDT 2012 Update 1 by Johan Arwidmark and Mikael Nystrom.  They’re the best at this stuff.  I bought it to update my own knowledge and to pick up some more on advanced concepts:

You can get this book on:

Take the time and have a look at what MDT can do for you.  If you have a lab it can be a time saver.  If you’re a consultant, it can be installed in a VM on Windows 8 or 8.1 Client Hyper-V and used on multiple client sites to rapidly provision servers (make sure you use the customer’s volume license product key).  MDT is free and powerful, and that makes it something worthwhile.

System Center 2012 R2 Is Required To Manage WinServ or Hyper-V 2012 R2

Part of me wishes Microsoft had not made Windows Server 2012 R2 available via MSDN/TechNet on Monday (I’ve had no confirmation of MVLS yet).  Why?  Confusion, that’s why.  It seems that lots of people are assuming things.  I hate when people assume.

You cannot manage any of the following using System Center 2012/SP1:

  • Windows Server 2012 R2
  • Windows Server 2012 R2 Hyper-V
  • Hyper-V Server 2012 R2

Yes, it was announced that System Center 2012 R2 will be out at the same time as Windows Server 2012 R2.  And it will be …. when both products are made generally available on October 18th as was announced.  That was the promise.  We got lucky because Microsoft put Windows Server out early via limited channels so the community could start testing and working on compatibility a little earlier than expected.  Windows Server 2012 R2 is still not generally available.  You cannot actually buy Windows Server 2012 R2 yet.

Please be patient.

Windows Server 2012 R2 & Windows 8.1 RTM Are Available On MSDN and TechNet

I have just confirmed that the news is true; Windows Server 2012 R2 and Windows 8.1 were in fact legitimately posted by Microsoft onto:

  • MSDN
  • And yes … TechNet

Microsoft blogged about it a little while ago, Mary Jo Foley posted about it, and I got an official email while here in the airport, on my way to London, where I’m demo-ing the Preview build Sad smile

I have personally logged into all 3 services and the media is there.  Woohoo!

Remember:

  • MSDN licenses are for test & dev
  • TechNet is for evaluation
  • The only legit production source is for those who have Software Assurance on their server licenses, and that source is MVLS …. this will not be updated until October 18th when Windows 8.1 and Windows Server 2012 R2 are generally available (GA – available to buy)

You cannot buy WS2012 R2 or Windows 8.1 yet.  Windows 8.1 Enterprise won’t be out for a little while.  System Center 2012 R2 (2012 does not support the new OS) is not out yet either – all will be out at the previously announced GA date.

The Effects Of WS2012 R2 Storage Spaces Write-Back Cache

In this post I want to show you the amazing effect that Write-Back Cache can have on the write performance of Windows Server 2012 R2 Storage Spaces.  But before I do, let’s fill in some gaps.

Background on Storage Spaces Write-Back Cache

Hyper-V, and many other applications/services/etc, does something called write-through.  In other words, it bypasses write caches of your physical storage.  This is to avoid corruption.  Keep this in mind while I move on.

In WS2012 R2, Storage Spaces introduces tiered storage.  This allows us to mix one tier of HDD (giving us bulk capacity) with one tier of SSD (giving us performance).  Normally a heap map process runs at 1am (task scheduler, and therefore customisable) and moves around 1 MB slices of files to the hot SSD tier or to the cold HDD tier, based on demand.  You can also pin entire files (maybe a VDI golden image) to the hot tier.

In addition, WS2012 R2 gives us something called Write-Back Cache (WBC).  Think about this … SSD gives us really fast write speeds.  Write caches are there to improve write performance.  Some applications are using write-through to avoid storage caches because they need the acknowledgement mean that the write really went to disk.

What if abnormal increases in write behaviour led to the virtual disk (a LUN in Storage Spaces) using it’s allocated SSD tier to absorb that spike, and then demote the data to the HDD tier later on if the slices are measured as cold.

That’s exactly what WBC, a feature of Storage Spaces with tiered storage, does.  A Storage Spaces tiered virtual disk will use the SSD tier to accommodate extra write activity.  The SSD tier increases the available write capacity until the spike decreases and things go back to normal.  We get the effect of a write cache, but write-through still happens because the write really is committed to disk rather than sitting in the RAM of a controller.

Putting Storage Spaces Write-Back Cache To The Test

What does this look like?  I set up a Scale-Out File Server that uses a DataOn DNS-1640D JBOD.  The 2 SOFS cluster nodes are each attached to the JBOD via dual port LSI 6 Gbps SAS adapters.  In the JBOD there is a tier of 2 * STEC SSDs (4-8 SSDs is a recommended starting point for a production SSD tier) and a tier of 8 * Seagate 10K HDDs.  I created 2 * 2-way mirrored virtual disks in the clustered Storage Space:

  • CSV1: 50 GB SSD tier + 150 GB HDD tier with 5 GB write cache size (WBC enabled)
  • CSV2: 200 GB HDD tier with no write cache (no WBC)

Note: I have 2 SSDs (sub-optimal starting point but it’s a lab and SSDs are expensive) so CSV1 has 1 column.  CSV2 has 4 columns.

Each virtual disk was converted into a CSV, CSV1 and CSV2.  A share was created on each CSV and shared as \Demo-SOFS1CSV1 and \Demo-SOFS1CSV2.  Yeah, I like naming consistency Smile

Then I logged into a Hyper-V host where I have installed SQLIO.  I configured a couple of params.txt files, one to use the WBC-enabled share and the other to use the WBC-disabled share:

  • Param1.TXT: \demo-sofs1CSV1testfile.dat 32 0x0 1024
  • Param2.TXT \demo-sofs1CSV2testfile.dat 32 0x0 1024

I pre-expanded the test files that would be created in each share by running:

  • "C:Program Files (x86)SQLIOsqlio.exe" -kW -s5 -fsequential -o4 –b64 -F"C:Program Files (x86)SQLIOparam1.txt"
  • "C:Program Files (x86)SQLIOsqlio.exe" -kW -s5 -fsequential -o4 -b64 -F"C:Program Files (x86)SQLIOparam2.txt"

And then I ran a script that ran SQLIO with the following flags to write random 64 KB blocks (similar to VHDX) for 30 seconds:

  • "C:Program Files (x86)SQLIOsqlio.exe" -BS -kW -frandom -t1 -o1 -s30 -b64 -F"C:Program Files (x86)SQLIOparam1.txt"
  • "C:Program Files (x86)SQLIOsqlio.exe" -BS -kW -frandom -t1 -o1 -s30 -b64 -F"C:Program Files (x86)SQLIOparam2.txt"

That gave me my results:

image

To summarise the results:

The WBC-enabled share ran at:

  • 2258.60 IOs/second
  • 141.16 Megabytes/second

The WBC-disabled share ran at:

  • 197.46 IOs/second
  • 12.34 Megabytes/second

Storage Spaces Write-Back Cache enabled the share on CSV1 to run 11.44 times faster than the non-enhanced share!!!  Everyone’s mileage will vary depending on number of SSDs versus HDDs, assigned cache size per virtual disk, speed of SSD and HDD, number of columns per virtual hard disk, and your network.  But one thing is for sure, with just a few SSDs, I can efficiently cater for brief spikes in write operations by the services that I am storing on my Storage Pool.

Credit: I got help on SQLIO from this blog post on MS SQL Tips by Andy Novick (MVP, SQL Server).