A blog covering Azure, Hyper-V, Windows Server, desktop, systems management, deployment, and so on …
Microsoft has launched an Excel-based sizing tool to help you plan Storage Spaces (Scale-Out File Server) and guidance on how to design your Storage Spaces deployments.
Here’s the sizing for a very big SOFS that will require 4 x SOFS server nodes and 4 x 60 disk JBODs:
The considerations guide will walk you through using the sizing tool.
Some updates are required – some newer disk sizes aren’t included – but this is a great starting point for a design process.
This post will walk you through figuring out Azure Storage accounts, pricing, and redundancy. One of the first things you will want to do when deploying virtual machines, websites (they can use them), or anything else in Azure is a storage account. But I find that most people, if they bother to search for the Azure storage pricing page, find the official page and get blizzard blindness.
I find these pages to be useful:
I imagine that you are the regular IT guy that asks for X GBs of a LUN at a certain RAID level. Things are different in Azure. A Storage Account is an access point to Azure storage. The account has a URL with permission-based access:
You create the storage account with a selected resiliency level. You do not configure a size – a single storage account can storage up to 500 TB of data and you pay for the amount of each service type that is contained within the storage account. You can create containers within a storage account that are similar to folders, allowing you to logically place your data/files. And there are remote access tools for managing storage.
So all this means that your storage “request” becomes … I want a storage account with name X (which is used as the basis for the URL) of resiliency Y in region Z.
There are 4 models of resiliency:
I know from experience that when you consider deploying a storage account for the first time you will be be wondering “how do I know where to put my files to be sure that I’m charged at the right rate?”. Don’t worry … all this is handled for you. Store your files and they are treated appropriately. Here are the 4 service types:
For the most part, we in the IaaS world are concerned with:
However, you might see other services being used for monitoring data, configuration files, metadata, etc. Don’t worry, that stuff is tiny, Azure storage is CHEAP, and Azure manages all that stuff.
Everything above is related to Standard Storage accounts. However, Azure offers the ability to deploy Premium Storage where virtual machine files are stored on SSDs instead of HDDs. This increases IOPS and reduces latency, and comes at a much greater cost. Don’t be foolish and deploy Premium Storage based on gut feeling – this cost is big enough that a proof-of-concept with realistic load simulation has determined that Standard Storage was insufficient.
The big news of the last 24 hours is that Windows 10 will be released on July 29th. I posted before The Verge, etc, that I will be away and not reporting on the release on that date.
It’s taken me nearly all day to fast-read through this lot. Here’s a dump of info from Build, Ignite, and since Ignite. Have a nice weekend!
You have probably already heard about Windows Insider, a program for providing feedback and shaping the future of Windows on client devices – not that I did not say “Windows 10” because the Insiders program will live beyond the RTM of Windows 10 this summer.
Similarly, the Windows Server group has launched a feedback forum. Here you can:
Now let’s be realistic – not everything will be done:
Here’s what I’ve voted for, commented on or submitted so far:
This is your opportunity to shape Windows Server. I’ve had that privilege as an MVP – it’s not always immediate but there are headline things in WS2016 that I’ve contributed some feedback for and it feels damned good to see them presented on stage. You can feel that too. If you choose to stay silent, then please stay that way when you’re unhappy.
Didier Van Hoye, myself and Carsten Rachfahl (all Hyper-V MVPs) were at Microsoft Ignite last week and we met up at the end to record a chat between the 3 of us, where we discussed some of our highlights from the conference. You can catch this video on the Hyper-V Amigos site.
Oh yeah, it was painful watching myself in this video 🙂 That was the last time Carsten will let me hold a microphone!
In this post I will show you how to set up a Scale-Out File Server using Windows Server 2016 Storage Spaces Direct (S2D). Note that:
Also note that:
Deploy at least 4 identical storage servers with WS2016. My lab consists of machines that have 4 DAS SAS disks. You can tier storage using SSD or NVMe, and your scalable/slow tier can be SAS or SATA HDD. There can be a max of tiers only: SSD/NVMe and SAS/SATA HDD.
Configure the IP addressing of the hosts. Place the two storage/cluster network into two different VLANs/subnets.
My nodes are Demo-S2D1, Demo-S2D2, Demo-S2D3, and Demo-S2D4.
You will need:
Here’s the PowerShell to do this:
Add-WindowsFeature –Name File-Services, Failover-Clustering –IncludeManagementTools
You can use -ComputerName <computer-name> to speed up deployment by doing this remotely.
It is good practice to do this … so do it. Here’s the PoSH code to validate a new S2D cluster:
You can use the GUI, but it’s a lot quicker to use PowerShell. You are implementing Storage Spaces so DO NOT ADD ELGIBLE DISKS. My cluster will be called Demo-S2DC1 and have an IP of 172.16.1.70.
New-Cluster -Name Demo-S2DC1 -Node Demo-S2D1, Demo-S2D2, Demo-S2D3, Demo-S2D4 -NoStorage -StaticAddress 172.16.1.70
There will be a warning that you can ignore:
There were issues while creating the clustered role that may prevent it from starting. For more information view the report file below.
You will probably use the default of dynamic quorum. You can either use a cloud witness (a storage account in Azure) or a file share witness, but realistically, Dynamic Quorum with 4 nodes and multiple data copies across nodes (fault domains) should do the trick.
The two cluster networks in my design will also be used for storage communications with the Hyper-V hosts. Therefore I need to configure these IPs for Client communications:
Doing this will also enable each server in the S2D SOFS to register it’s A record of with the cluster/storage NIC IP addresses, and not just the management NIC.
This is not on by default. You enable it using PowerShell:
(Get-Cluster).DASModeEnabled=1
Open up FCM and connect to the cluster. You’ll notice lots of stuff in there now. Note the new Enclosures node, and how each server is listed as an enclosure. You can browse the Storage Spaces eligible disks in each server/enclosure.
I then create a pool called Pool1 on the cluster Demo-S2DC1 using PowerShell – this is because there are more options available to me than in the UI:
New-StoragePool -StorageSubSystemName Demo-S2DC1.demo.internal -FriendlyName Pool1 -WriteCacheSizeDefault 0 -FaultDomainAwarenessDefault StorageScaleUnit -ProvisioningTypeDefault Fixed -ResiliencySettingNameDefault Mirror -PhysicalDisk (Get-StorageSubSystem -Name Demo-S2DC1.demo.internal | Get-PhysicalDisk)
Get-StoragePool Pool1 | Get-PhysicalDisk |? MediaType -eq SSD | Set-PhysicalDisk -Usage Journal
The you create the CSVs that will be used to store file shares in the SOFS. Rules of thumb:
Using this PoSH you will lash out your CSVs in no time:
$CSVNumber = “4”
$CSVName = “CSV”
$CSV = “$CSVName$CSVNumber”
New-Volume -StoragePoolFriendlyName Pool1 -FriendlyName $CSV -PhysicalDiskRedundancy 2 -FileSystem CSVFS_REFS –Size 200GB
Set-FileIntegrity “C:\ClusterStorage\Volume$CSVNumber” –Enable $false
The last line disables ReFS integrity streams to support the storage of Hyper-V VMs on the volumes. You’ll see from the screenshot what my 4 node S2D SOFS looks like, and that I like to rename things:
Note how each CSV is load balanced. SMB redirection will redirect Hyper-V hosts to the owner of a CSV when the host is accessing files for a VM that is stored on that CSV. This is done for each VM connection by the host using SMB 3.0, and ensures optimal flow of data with minimized/no redirected IO.
There are some warnings from Microsoft about these volumes:
Oops! This is a technical preview and this should be pure lab work that your willing to lose.
The purpose of this post is to create a SOFS from the S2D cluster, with the sole purpose of the cluster being to store Hyper-V VMs that are accessed by Hyper-V hosts via SMB 3.0. If you are building a hyperconverged cluster (not supported by the current TPv2 preview release) then you stop here and proceed no further.
Each of the S2D cluster nodes and the cluster account object should be in an OU just for the S2D cluster. Edit the advanced security of the OU and grand the cluster account object create computer object and delete compute object rights. If you don’t do this then the SOFS role will not start after this next step.
Next, I am going to create an SOFS role on the S2D cluster, and call it Demo-S2DSOFS1.
New-StorageFileServer -StorageSubSystemName Demo-S2DC1.demo.internal -FriendlyName Demo-S2DSOFS1 -HostName Demo-S2DSOFS1Create a -Protocols SMB
Create 1 share per CSV. If you need more shares then create more CSVs. Each share needs the following permissions:
You can use the following PoSH to create and permission your shares. I name the share folder and share name after the CSV that it is stored on, so simply change the $ShareName variable to create lots of shares, and change the permissions as appropriate.
$ShareName = “CSV1”
$SharePath = “$RootPath\$ShareName\$ShareName”
md $SharePath
New-SmbShare -Name $ShareName -Path $SharePath -FullAccess Demo-Host1$, Demo-Host2$, Demo-HVC1$, “Demo\Hyper-V Admins”
Set-SmbPathAcl -ShareName $ShareName
On your hosts/clusters create VMs that store all of their files on the path of the SOFS, e.g. \\Demo-S2DSOFS1\CSV1\VM01, \\Demo-S2DSOFS1\CSV1\VM02, etc.
This post was written not long after the release of TPv2:
This session, presented by Claus Joergensen, Michael Gray, and Hector Linares can be found on Channel 9:
This design is known as converged (not hyper-converged). There are two tiers:
The storage tier or SOFS has two layers:
System Center is an optional management layer.
Note: you might hear/see/read the term SSDi (there’s an example in one of the demos in the video). This was an old abbreviation. The correct abbreviation for Storage Spaces Direct is S2D.
The focus of this talk is the storage tier. S2D collapses this tier so that there is no need for a SAS layer. Note, though, that the old SOFS design continues and has scenarios where it is best. S2D is not a replacement – it is another design option.
S2D can be used to store VM files on it. It is made of servers (4 or more) that have internal or DAS disks. There are no shared JBODs. Data is mirrored across each node in the S2D cluster, therefore the virtual disks/CSVs are mirrored across each node in the S2D cluster.
S2D introduces support for new disks (with SAS disks still being supported):
Other features:
You have two options for deploying S2D. Windows Server 2016 will introduce a hyper-converged design – yes I know; Microsoft talked down hyper-convergence in the past. Say bye-bye to Nutanix. You can have:
Hyper-convergence is being tested now and will be offered in a future release of WS2016.
As I said, shared JBOD SOFS continues as a deployment option. In other words, an investment in WS2012 R2 SOFS is still good and support is continued. Node that shared JBODs offer support for dual parity virtual disks (for archive data only – never virtual machines).
S2D adds support for the cheapest of disks and the fastest of disks.
This is a conceptual, not an architectural, diagram.
A software storage bus replaces the SAS shared infrastructure using software over an Ethernet channel. This channel spans the entire S2D cluster using SMB 3.0 and SMB Direct – RDMA offers low latency and low CPU impact.
On top of this bus that spans the cluster, you can create a Storage Spaces pool, from which you create resilient virtual disks. The virtual disk doesn’t know that it’s running on DAS instead of shared SAS JBOD thanks to the abstraction of the bus.
File systems are put on top of the virtual disk, and this is where we get the active/active CSVs. The file system of choice for S2D is ReFS. This is the first time that ReFS is the primary file system choice.
Depending on your design, you either run the SOFS role on the S2D cluster (converged) or you run Hyper-V virtual machines on the S2D cluster (hyper-converged).
System Center is an optional management layer.
Data is stored in the form of extents. Each extent is 1 GB in size so a 100 GB virtual disk is made up of 100 extents. Below is an S2D cluster of 5 nodes. Note that extents are stored evenly across the S2D cluster. We get resiliency by spreading data across each node’s DAS disks. With 3-way mirroring, each extent is stored on 3 nodes. If one node goes down, we still have 2 copies, from which data can be restored onto a different 3rd node.
Note: 2-way mirroring would keep extents on 2 nodes instead of 3.
Extent placement is rebalanced automatically:
How we get scale-out and resiliency:
This is why we need good networking for S2D: RDMA. Forget your 1 Gbps networks for S2D.
You can also remove a system: rebalance it, and shrink extents down to fewer nodes.
Scale for TPv2:
S2D is fault tolerance to disk enclosure failure and server failure. It is resilient to 2 servers failing and cluster partitioning. The result should be uninterrupted data access.
Each S2D server is treated as the fault domain by default. There is fault domain data placement, repair and rebalancing – means that there is no data loss by losing a server. Data is always placed and rebalanced to recognize the fault domains, i.e. extents are never stored in just a single fault domain.
If there is a disk failure, there is automatic repair to the remaining disks. The data is automatically rebalanced when the disk is replaced – not a feature of shared JBOD SOFS.
If there is a temporary server outage then there is a less disruptive automatic data resync when it comes back online in the S2D cluster.
When there is a permanent server failure, the repair is controlled by the admin – the less disruptive temporary outage is more likely so you don’t want rebalancing happening then. In the event of a real permanent server loss, you can perform a repair manually. Ideally though, the original machine will come back online after a h/w or s/w repair and it can be resynced automatically.
Note that S2D uses ReFS (pronounced as Ree-F-S) as the file system of choice because of scale, integrity and resiliency:
No need for chkdsk. There is no disruptive offline scanning in ReFS:
Efficient VM checkpoints and backup:
Accelerated Fixed VHD/X Creation:
Yay! I wonder how many hours of my life I could take back with this feature?
Dynamic VHDX Expansion
Demo 1:
2 identical VMs, one on NTFS and one on ReFS. Both have 8 GB checkpoints. Deletes the checkpoint from the ReFS VM – The merge takes about 1-2 seconds with barely any metrics increase in PerfMon (incredible improvement). Does the same on the NTFS VM and … PerfMon shows way more activity on the disk and the process will take about 3 minutes.
Demo 2:
Next he creates 15 GB fixed VHDX files on two shares: one on NTFS and one on ReFS. ReFS file is created in less than a second while the previous NTFS merge demo is still going on. The NTFS file will take …. quite a while.
Demo 3:
Disk Manager open on one S2D node: 20 SATA disks + 4 Samsung NVMe disks. The S2D cluster has 5 nodes. There is a total of 20 NVMe devices in the single pool – a nice tidy aggregation of PCIe capacity. The 5 node is new so no rebalancing has been done.
Lots of VMs running from a different tier of compute. Each VM is running DiskSpd to stress the storage. But distributed Storage QoS is limiting the VMs to 100 IOPs each.
Optimize-StoragePool –FriendlyName SSDi is run to bring the 5th node into the cluster (called SSDi) online by rebalancing. Extents are remapped to the 5th node. The system goes full bore to maximize IOPS – but note that “user” operations take precedence and the rebalancing IOPS are lower priority.
Management is provided by SCOM and SCVMM. This content is focused on S2D, but the management tools also work with other storage options:
Roles:
Note: You can also use Hyper- Replica with/without ASR.
Demo:
He starts the process of bare-metal provisioning a SOFS cluster from VMM – consistent with the Hyper-V host deployment process. This wizard offers support for DAS or shared JBOD/SAN; this affects S2D deployment and prevents unwanted deployment of MPIO. You can configure existing servers or deploy a physical computer profile to do a bare-metal deployment via BMCs in the targeted physical servers. After this is complete, you can create/manage pools in VMM.
File server nodes can be added from existing machines or bare-metal deployment. The disks of the new server can be added to the clustered Storage Spaces pool. Pools can be tiered (classified). Once a pool is created, you can create a file share – this provisions the virtual disk, configures CSV, and sets up the file system for you – lots of automation under the covers. The wizard in VMM 2016 includes resiliency and tiering.
Right now, SCOM must do all the work – gathering a data from a wide variety of locations and determining health rollups. There’s a lot of management pack work there that is very hardware dependent and limits extensibility.
Microsoft reimagined monitoring by pushing the logic back into the storage system. The storage system determines health of the storage system. Three objects are reported to monitoring (PowerShell, SCOM or 3rd party, consumable through SMAPI):
Alerts will be remediated automatically where possible. The system automatically detects the change of health state from error to healthy. Updates to external monitoring takes seconds. Alerts from the system include:
Demo:
One of the cluster nodes is shut down. SCOM reports that a node is missing – there isn’t additional noise about enclosures, disks, etc. The subsystem abstracts that by reporting the higher error – that the server is down. The severity is warning because the pool is still online via the rest of the S2D cluster. The priority is high because this server must be brought back online. The server is restarted, and the alert remediates automatically.
Storage Spaces/JBODs has proven that you cannot use just any hardware. In my experience, DataON stuff (JBOD, CiB, HGST SSD and Seagate HDD) is reliable. On the other hand, SSDs by SanDisk are shite, and I’ve had many reports of issues with Intel and Quanta Storage Spaces systems.
There will be prescriptive configurations though partnerships, with defined platforms, components, and configuration. This is a work in progress. You can experiment with Generation 2 VMs.
I really hope that we don’t see OEMs creating “bundles” like they did for pre-W2008 clustering that cost more than the sum of the otherwise-unsupported individual components. Heck, who am I kidding – of course they will do that!!! That would be the kiss of death for S2D.
Demo Video:
They have two systems connected to a 4 node S2D cluster, with a sum total of 1.2 million 4K IOPS with below 1 millisecond latency, thanks to (affordable) SATA SSDs and Mellanox ConnectX-3 RDMA networking (2 x 40 Gbps ports per client). They remove RDMA from each client system. IOPS is halved and latency increases to around 2 milliseconds. RDMA is what enables low latency and low CPU access to the potential of the SSD capacity of the storage tier.
Hint: the savings in physical storage by using S2D probably paid for the networking and more.
Microsoft recorded and shared a video of my session, The Hidden Treasures of Windows Server 2012 R2 Hyper-V, along with the slides.
My second session, End to-End Azure Site Recovery Solutions for Small-Medium Enterprises in one of the community theatres, was not recorded so I have placed the slides up on slideshare.
Speaker: Net Pyle.
Answer: McDonalds running out of food at Ignite. But I digress … you lose your entire server room or data centre.
Hurricane Sandy wiped out Manhattan. Lots of big hosting facilities went offline. Some stayed partially online. And a handful stayed online.
Synchronous replication between cities. Asynchronous replication between countries. Not just about disaster recovery but also disaster avoidance.
It is volume based. Uses SMB 3.1.1. Works with any Windows data volume. Any fixed disk storage: iSCSI, Spaces, local disk or any storage fabric (iSCSI, FCoE, SAS, etc). You manage it using FCM (does not require a cluster), PowerShell, WMI, and in the future: Azure Site Recovery (ASR).
This is a feature of WS2016 and there is no additional licensing cost.
A demo that was done before, using a 2 node cluster, file changes in a VM in site A, replicates, and change shows up after failover.
Replicate one volume to another on the same server. Then move these disks to another server and use them as a seed for replication.
Block based replication. It is not DFS-R. Replication is done way down low. It is unaware of the concept of files so doesn’t know that they are used. It only cares about write IO. Works with CSVFS, NTFS and ReFS.
2 years of work by 10 people to create a disk filter driver that sits between the Volume Manager and the Partition Manager.
A log is kept of each write on primary server. The log is written through to the disk The same log is kept on the secondary site. The write is sent to the log in parallel on both sites. Only when the secondary site has written to the log in both sites is the write acknowledged
The write goes to the log on site A and acknowledged. Continuous replication sends the write to the log in the secondary site. Not interval based.
RDMA/SMB Direct can be used long range with Mellanox InfiBand Metro-X and Chelsio iWarp can do long distance. MSFT have tested 10KM, 25 KM, and 40KM networks to test this. Round trip latencies are hundreds of microseconds for 40 KM one-way (very low latency). SMB 3.1.1 has optimized built-in encryption. They are still working on this and you should get to the point where you want encryption on all the time.
Firewall: SMB and WS-MAN
Latency not an issue. Log volume recommendations are the same as above.
Test-SRTopology cmdlet. Checks requirements and recommendations for bandwidth, log sizes, IPS, etc. Runs for specified duration to analyse a potential source server for sizing replication. Run it before configuration replication against a proposed source volume and proposed destination.
Async crash consistency versus application consistency. Guarantee mountable volume. App must guarantee a usable file
Can replicate VSS snapshots.
You cannot use the replica volume. In this release they only do 1:1 replication, e.g. 1 node to 1 node, 1 cluster to 1 cluster, and 1 half cluster to another half cluster. You cannot do legs of replication.
You can do Hyper-V Replica from A to B and SR from B to C.
Resizing replicated volumes interrupts replication. This might change – feedback.
Latest drivers. Most problems are related to drivers, not SR. Filter drivers can be dodgy too.
Understand your performance requirements. Understand storage latency impact on your services. Understand network capacity and latency. PerfMon and DiskSpd are your friends. Test workloads before and after SR.
In a VM. Requires WS2016 DC edition. Work on any hypervisor. It works in Azure, but no support statement yet.
HVR understands your Hyper-V workload. It works with HTTPS and certificates. Also in Std edition.
SR offers synchronous replication. Can create stretched guest clusters. Can work in VMs that are not in Hyper-V.
Lots of reasons to use SQL AGs. SR doesn’t require SQL Ent. Can replicate VMs at host volume level. SR might be easier than SQL AGs. You must use write ordering/consistency if you use any external replication of SQL VMs – includes HVR/ASR.
It is a DR solution with lots of bandwidth between them.
Not for stretch-cluster SOFS – you’d do cluster-to-cluster replication for that.
Runs DiskSpd on volume on source machine.
In this demo, the 2 machines were 25 KM apart with an iWarp link. Replaced this with fibre and did 60,000 IOPS.
Requires SCVMM. You get end-end orchestration. Groups VMs to replicate together. Supports for Azure Automation runbooks. Support for planned/unplanned failover. Preview in July/August.
Hidden volume.
A guide was published on Monday on TechNet.
Questions to srfeed <at> microsoft.com
