Speakers: Joshua Adams and Jason Gerend, Microsoft.
Designing a Storage Spaces Solution
- Size your disks for capacity and performance
- size your storage enclosures
- Choose how to handlw disk failures
- Pick the number of cluster nodes
- Select a hardware solution
- Design your storage pools
- Design your virtual disks
Size your disks – for capacity (HDDs)
- Identify your workloads and resiliency type: Parity for backups and mirror for everything else.
- Estimate how much raw capacity you need. Currently capcity x% data grown X data copies (if your using mirrors). Add 12% initially for automatic virtual disk repairs and meta data overhead. Example: 135 TB x 1. x 3 data copies + 12 % = 499 TB raw capacity
- Size your HDDs: Pick big 7200 RPM NL SAS HDDs. Fast HDD not required is using SSD tier.
Software Defined Storage Calculator allows you to size and design a deployment and it generates the PowerShell. Works with WS2012 R2 and WS2016, disaggregated and hyperconverged deployments.
Size your disks – for performance (SSDs)
- How many SSDs to use. Sweet spot is 1 SSD for every 2-4 HDDs. Typically 4-5 SSDs per enclosure per pool. More SSDs = more absolute performance
- Determine the SD size. 800 GB SSDs are typical. Larger SSD capacity = can handle larger amounts of active data. Anticipate around 10% of SSD capacity for automatically repairing after an SSD failure.
Example 36 x 800 GB SSDs.
Size you Enclosures
- Pick the enclosure size (12, 24, 60, etc disks)
- Pick the number of enclosures. If you have 3 or 4 then you have enclosure awareness/fault tolerance, depending on type of mirroring.
- Each enclosure should have an identical number of disks.
Example, 3 x 60 bay JBODs each with 48 HDDs and 12 SSDs
The column count is fixed between 2 tiers. The smaller tier (SSD) limits the column count. 3-4 columns is a sweet spot.
Expanding pools has an overhead. Not trivial but it works. Recommend that you fill JBODs.
Choose how to Handle Disk Failures
- Simultaneous disk failures to tolerate. Use 2 data copies for small deployments and disks, and/or less important data. use 3 data copies for larger deployments and disks, and for more important data.
- Plan to automatically repair disks. Instead of hot spares, set aside pool capacity to automatically replace failed disks. Also effects column count … more later.
Example: 3-way mirrors.
Pick the number of Cluster Nodes
Start with 1 node per enclosure and scale up/down depending on the amount of compute required. This isn’t about performance; it’s about how much compute you can afford to lose and still retain HA.
Example: 3 x 3 = 3 SOFS nodes + 3 JBODs.
Select a hardware vendor
- RAID Inc
- Microsoft/Dell CPS
Design your Storage Pools
- Management domains: put your raw disks in the pool and manage them as a group. Some disk settings are applied at the pool level.
- More pools = more to manage. Pools = fault domains. More pools = less risk – increased resiliency and resiliency overhead..
Start with 84 disks per pool.
Divide disks evenly between pools.
Design your Virtual Disks
- Where storage tiers, write-back cache and enclosure awareness are set.
- More VDs = more uniform load balancing, but more to manage.
- This is where column count come in. More columns = more throughput, but more latency. 3-4 columns is best.
- Load balancing is dependent on identical virtual disks.
- To automatically repair after a disk failure, need at least one more disk per tier than columns for the smallest tier, which is usually the SSD tier.
- Set aside 10% of SSD and HDD capacity for repairs.
- Start with 2 virtual disks per node.
- Add more to keep virtual disk size to 10 TB or less. Divide SSD and HDD capacity evenly between virtual disks. Use 3-4 columns if possible.
Best Practices for WS2012 R2
- Scale by adding fully populated clusters. Get used to the concept of storage/compute/networking stamps.
- Monitor your existing workloads for performance. The more you know about the traits of your unique workloads, the better future deployments will be.
- Do a PoC deployment. Use DiskSpd and fault injection to stress the solution. Monitor the storage tiers performance to determine how much SSD capacity you need to fit a given scale of your workloads into SSD tiers.
WORK WITH A TRUSTED SOLUTION VENDOR. Not all hardware is good, even if it is on the HCL. Some are better than others, and some suck. In my opinion Intel and Quanta suck. DataON is excellent. Dell appears to have gone through hell during CPS development to be OK. And some disks, e.g. SanDISK, are the spawn of Satan, in my experience – Note that Dell use SanDISK and Toshiba so demand Toshiba only SSDs from Dell. HGST SSDs are excellent.
Deployment Best Practices
- Disable TRIM on SSDs. Some drives degrade performance with TRIM enabled.
- Disable all disk based caches – if enabled if degrades performance when write-through is used (Hyper-V).
- Use LB (least blocks) for MPIO policy. For max performance, set individual SSDs to Round Robin. This must be done on each SOFS node.
- Optimize Storage Spaces repair settings on SOFS. Use Fast Rebuild. Change it from Auto to Always on the pool. This means that 5 minutes after a write failure, a rebuild will automatically start. Pulling a disk does not trigger an automatic rebuild – an expensive process.
- Install the latest updates. Example: repair process got huge improvement in November 2014 update.
Deployment & Management Best Practices
- Deploy using VMM or PowerShell. FCM is OK for small deployments.
- VMM is great for some stuff, but in 2012 R2 it doesn’t do tiering etc. It can create the cluster well and manage shares, but for disk creation, use PowerShell.
- Monitor it using SCOM with the new Storage Spaces management pack.
- Also use Test-StorageHealth.PS1 to do some checks occasionally. It needs tweaking to size it for your configuration.
Design Closing Thoughts
- Storage Spaces solutions offer: 2-4 cluster nodes and 1-4 JBODs. Store 100 to as many as 2000 VMs.
- Storage Pool Design; HDDs provide most of the capacity. SSDs offer performance. Up to 84 disks per pool.
- Virtual Disk design: Set aside 10% of SSD and HDD capacity for repairs. Start with 2 VDs per node. Max 0 TB/virtual disk. 3-4 volums for balanced performance.
Coming in May
- Storage Spaces Design Considerations Guide (basis of this presentation)
- Storage Spaces Design Calculator (spreadsheet used in this presentation)