{"id":12167,"date":"2012-03-07T18:53:19","date_gmt":"2012-03-07T18:53:19","guid":{"rendered":"https:\/\/aidanfinn.com\/?p=12167"},"modified":"2012-03-07T18:53:19","modified_gmt":"2012-03-07T18:53:19","slug":"windows-server-8-hyper-v-storage-strategies","status":"publish","type":"post","link":"https:\/\/aidanfinn.com\/?p=12167","title":{"rendered":"Windows Server 2012 Hyper-V Storage Strategies"},"content":{"rendered":"<p><em>This article was written just after the beta of WS2012 was launched.\u00a0 We now now that the performance of SMB 3.0 is -really- good, e.g. 1 million IOPS from a VM good.<\/em><\/p>\n<p>WS2012 is bringing a lot of changes in how we design storage for our Hyper-V hosts.\u00a0 There\u2019s no one right way, just lots of options, which give you the ability to choose the right one for your business.<\/p>\n<p>There were two basic deployments in Windows Server 2008 R2 Hyper-V, and they\u2019re both sill valid with Windows Server 2012 Hyper-V:<\/p>\n<ul>\n<li>Standalone: The host had internal disk or DAS and the VMs that ran on the host were stored on this disk.<\/li>\n<li>Clustered: You required a SAN that was either SAS, iSCSI, or FIbre Channel (FC) attached (as below).<\/li>\n<\/ul>\n<p><a href=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/03\/image1.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; padding-left: 0px; padding-right: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-top: 0px; border-width: 0px;\" title=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/03\/image_thumb1.png\" border=\"0\" alt=\"image\" width=\"404\" height=\"364\" \/><\/a><\/p>\n<p>And there\u2019s the rub.\u00a0 Everyone wants VM mobility and fault tolerance.\u00a0 I\u2019ve talked about some of this in recent posts.\u00a0 Windows Server 2012 Hyper-V has <a href=\"https:\/\/aidanfinn.com\/?p=12136\" target=\"_blank\">Live Migration that is independent of Failover Clustering<\/a>.\u00a0 Guest clustering is limited to iSCSI in Windows Server 2012 Hyper-V but Windows Server 2012 Hyper-V is adding support for <a href=\"https:\/\/aidanfinn.com\/?p=12115\" target=\"_blank\">Virtual Fibre Channel<\/a>.<\/p>\n<p>Failover Clustering is still the ideal.\u00a0 Whereas Live Migration gives proactive migration (move workloads before a problem, e,g, to patch a host), Failover Clustering provides high availability via reactive migration (move workloads automatically in advance of a problem, e.g. host failure).\u00a0 The problem here is that a cluster requires shared storage.\u00a0 And that has always been expensive iSCSI, SAS, or FC attached storage.<\/p>\n<p>Expensive?\u00a0 To whom?\u00a0 Well, to everyone.\u00a0 For most SMEs that buy a cluster, the SAN is probably the biggest IT investment that that company will ever make.\u00a0 Wouldn\u2019t it suck if they got it wrong, or if they had to upgrade\/replace it in 3 years?\u00a0 What about the enterprise?\u00a0 They can afford a SAN.\u00a0 Sure, but their storage requirements keep growing and growing.\u00a0 Storage is not cheap (don\u2019t dare talk to me about $100 1 TB drives).\u00a0 Enterprises are sick and tired of being held captive by the SAN companies for 100% of their storage needs.<\/p>\n<p>We\u2019re getting new alternatives from Microsoft in Windows Server 2012.\u00a0 This is all made possible by a new version of the SMB protocol.<\/p>\n<p><strong><span style=\"text-decoration: underline;\">SMB 3.0 (Formerly SMB 2.2)<\/span><\/strong><\/p>\n<p>Windows Server 2012 is bringing us a new version of the SMB protocol.\u00a0 With the additional ability to do multichannel, where file share data transfer automatically spans multiple NICs with fault tolerance, we are now getting support to store virtual machines on a file server, as long as both client (Hyper-V host) and server (file server) are running Windows Server 2012 or above.<\/p>\n<p>If you\u2019re thinking ahead then you\u2019ve already started to wonder about how you will backup these virtual machines using an agent on the host.\u00a0 The host no longer has \u201cdirect\u201d access to the VMs as it would with internal disk, DAS, or a SAN.\u00a0 Windows Server 2012 VSS appears to be quite clever, intercepting a backup agents request to VSS snapshot a file server stored VM, and redirecting that to VSS on the file server.\u00a0 We\u2019re told that this should all be transparent to the backup agent.<\/p>\n<p>Now we get some new storage and host design opportunities.<\/p>\n<p><strong><span style=\"text-decoration: underline;\">Shared File Server \u2013 No Hyper-V Clustering<\/span><\/strong><\/p>\n<p>In this example a single Windows Server 2012 file server is used to store the Hyper-V virtual machines.\u00a0 The Hyper-V hosts can use the same file server, and they are not clustered.\u00a0 With this architecture, you can do <a href=\"https:\/\/aidanfinn.com\/?p=12136\" target=\"_blank\">Live Migration between the two hosts, even without a cluster<\/a>.<\/p>\n<p><a href=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/03\/image2.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; padding-left: 0px; padding-right: 0px; display: inline; padding-top: 0px; border-width: 0px;\" title=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/03\/image_thumb2.png\" border=\"0\" alt=\"image\" width=\"304\" height=\"275\" \/><\/a><\/p>\n<p>What about performance?\u00a0 SMB is going to suck, right?\u00a0 Not so fast, my friend!\u00a0 Even with a pair of basic 1 Gbps NICs for SMB 3.0 traffic (instead of a pair of NICs for iSCSI), I\u2019ve been told that you can expect iSCSI-like speeds, <em>and maybe <\/em>even better.\u00a0 At 10 Gbps \u2026 well <img decoding=\"async\" class=\"wlEmoticon wlEmoticon-smile\" style=\"border-style: none;\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/03\/wlEmoticon-smile5.png\" alt=\"Smile\" \/> The end result is cheaper and easier to configure storage.<\/p>\n<p>With the lack of fault tolerance, this deployment type is probably suitable only for small businesses and lab environments.<\/p>\n<p><strong><span style=\"text-decoration: underline;\">Scale Out File Server (SOFS) \u2013 No Hyper-V Clustering<\/span><\/strong><\/p>\n<p>Normally we want our storage to be fault tolerant. That\u2019s because all of our VMs are probably on that single SAN (yes, some have the scale and budget for spanning SANs but that\u2019s a whole different breed of organisation).\u00a0 Normally we would need a SAN made up fault tolerant disk tray$, switche$, controller$, hot $pare disk$, and $o on.\u00a0\u00a0 I think you get the point.<\/p>\n<p>Thanks to the innovations of Windows Server 2012, we\u2019re going to get a whole new type of fault tolerant storage called a SOFS.<\/p>\n<p><a href=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/03\/image3.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; padding-left: 0px; padding-right: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-top: 0px; border-width: 0px;\" title=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/03\/image_thumb3.png\" border=\"0\" alt=\"image\" width=\"304\" height=\"373\" \/><\/a><\/p>\n<p>What we have in a SOFS is an active\/active file server cluster.\u00a0 The hosts that store VMs on the cluster use UNC paths instead of traditional local paths (even for CSV).\u00a0 The file servers in the SOFS cluster work as a team.\u00a0 A role in SMB 3.0 called the witness runs on the Hyper-V host (SMB witness client) and file server (SMB witness server).\u00a0 With some clever redirection the SOFS can handle:<\/p>\n<ul>\n<li>Failure of a file server with just a blip in VM I\/O (no outage).\u00a0 The cluster will allow the new host of the VMs to access the files without a 60 second delay you might see in today\u2019s technology.<\/li>\n<li>Live Migration of a VM from one host to another with a smooth transition of file handles\/locks.<\/li>\n<\/ul>\n<p>And VSS works through the above redirection process too.<\/p>\n<p>One gotcha: you might look at this and this this is a great way to replace current file servers.\u00a0 The SOFS is intended only for large files with little metadata access (few permissions checks, etc).\u00a0 The currently envisioned scenarios are SQL Server file storage and Hyper-V VM file storage.\u00a0 End user file shares, on the other hand, feature many small files with lots of metadata access and are not suitable for SOFS.<\/p>\n<p>Why is this?\u00a0 To make the file servers active\/active with smooth VM file handle\/lock transition, the storage that the file servers are using consists of 1 or more Cluster Shared Volumes (CSVs).\u00a0 This uses CSV v2.0, not the version we have in Windows Server 2008 R2.\u00a0 The big improvements in CSV 2.0 are:<\/p>\n<ul>\n<li>Direct I\/O for VSS backup<\/li>\n<li>Concurrent backup across all nodes using the CSV<\/li>\n<\/ul>\n<p>Some activity in a CSV does still cause redirected I\/O, and an example of that is metadata lookup.\u00a0 Now you get why this isn\u2019t good for end user data.<\/p>\n<p>When I\u2019ve talked about SOFS many have jumped immediately to think that it was only for small businesses.\u00a0 Oh you fools!\u00a0 Never assume!\u00a0 Yes, SOFS can be for the small business (more later).\u00a0 But where this really adds value is that larger business that feels like they are held hostage by their SAN vendors.\u00a0 Organisations are facing a real storage challenge today.\u00a0 SANs are not getting cheaper, and the storage scale requirements are rocketing.\u00a0 SOFS offers a new alternative.\u00a0 For a company that requires certain hardware functions of a SAN (such as replication) then SOFS offers an alternative tier of storage.\u00a0 For a hosting company where every penny spent is a penny that makes them more expensive in the yes of their customers, SOFS is a fantastic way to provide economic, highly performing, scalable, fault tolerant storage for virtual machine hosting.<\/p>\n<p>The SOFS cluster does require shared storage of some kind.\u00a0 It can be made up of the traditional SAN technologies such as SAS, iSCSI, or Fibre Channel with the usual RAID suspects.\u00a0 Another new technology, called PCI RAID, is on the way.\u00a0 It will allow you to use just a bunch of disks (JBOD) and you can have fault tolerance in the form of mirroring or parity (Windows Server 2012 Storage Spaces and Storage Pools).\u00a0 It should be noted that if you want to create a CSV on a Storage Space then it must use mirroring, and not parity.<\/p>\n<p><em>Update: I had previously blogged in this article that I was worried that SOFS was suitable only for smaller deployments.\u00a0 I was seriously wrong.<\/em><\/p>\n<p>Good news for those small deployment: Microsoft is working with hardware partners to create a cluster-in-a-box (CiB) architecture with 2 file servers, JBOD and PCI RAID.\u00a0 Hopefully it will be economic to acquire\/deploy.<\/p>\n<p><em>Update: And for the big biz that needs big IOPS for LOB apps, there are CiB solutions for you too, based on Infiniband networking, RDMA (SMB Direct), and SSD, e.g. a 5U appliance having the same IOPS are 4 racks of fibre channel disk.<\/em><\/p>\n<p>Back to the above architecture, I see this one being useful in a few ways:<\/p>\n<ul>\n<li>Hosting companies will like it because every penny of each Hyper-V host is utilised.\u00a0 Having N+1 or N+2 Hyper-V hosts means you have to add cost to your customer packages and this makes you less competitive.<\/li>\n<li>Larger enterprises will want to reduce their every-5 year storage costs and this offers them a different tier of storage for VMs that don\u2019t require those expensive SAN features such as LUN replication.<\/li>\n<\/ul>\n<p><strong><span style=\"text-decoration: underline;\">SOFS \u2013 Hyper-V Cluster<\/span><\/strong><\/p>\n<p>This is the next step up from the previous solution.\u00a0 It is a fully redundant virtualisation and storage infrastructure without the installation of a SAN.\u00a0 A SOFS (active-active file server cluster) provides the storage.\u00a0 A Hyper-V cluster provides the virtualisation for HA VMs.<\/p>\n<p><a href=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/03\/image4.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone\" style=\"background-image: none; padding-left: 0px; padding-right: 0px; display: inline; padding-top: 0px; border-width: 0px;\" title=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/03\/image_thumb4.png\" border=\"0\" alt=\"image\" width=\"304\" height=\"373\" \/><\/a><\/p>\n<p>The Hyper-V hosts are clustered.\u00a0 If they were direct attached to a SAN then they would place their VMs directly on CSVs.\u00a0 But in this case they store their VMs on a UNC path, just as with the previous SMB 3.0 examples.\u00a0 VMs are mobile thanks to Live Migration (as before without Hyper-V clusters) and thanks to Failover.\u00a0 Windows Server 2012 Clustering has had a lot of work done to it; my favourite change being Cluster Aware Updating (easy automated patching of a cluster via Automatic Updates).<\/p>\n<p>The next architectures \u201cup\u201d from this one are Hyper-V clusters that use SAS, iSCSI, or FC.\u00a0 Certainly SOFS is going to be more scalable than a SAS cluster.\u00a0 I\u2019d also argue that it could be more scalable than iSCSI or FC purely based on cost.\u00a0 Quality iSCSI or FC SANs can do things at the hardware layer that a file server cluster cannot, but you can get way more fault tolerant storage per Euro\/Dollar\/Pound\/etc with SOFS.<\/p>\n<p>So those are your options \u2026 in a single site <img decoding=\"async\" class=\"wlEmoticon wlEmoticon-smile\" style=\"border-style: none;\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/03\/wlEmoticon-smile5.png\" alt=\"Smile\" \/><\/p>\n<p><strong><span style=\"text-decoration: underline;\">What About Hyper-V Cluster Networking? Has It Changed?<\/span><\/strong><\/p>\n<p>In a word: no.<\/p>\n<p>The basic essentials of what you need are still the same:<\/p>\n<ol>\n<li>Parent\/management networking<\/li>\n<li>VM connectivity<\/li>\n<li>Live Migration network (this should usually be your first 10 GbE network)<\/li>\n<li>Cluster communications network (heartbeat and redirected IO which does still have a place, even if not for backup)<\/li>\n<li>Storage 1 (iSCSI or SMB 3.0)<\/li>\n<li>Storage 2 (iSCSI or SMB 3.0)<\/li>\n<\/ol>\n<p><em>Update: We now have two types of redirected IO that both support SMB Multichannel and SMB Direct.\u00a0 SMB redirection (high level) is for those short metadata operations, an block level redirect (2x faster) is for sustained redirection IO operations such as a storage path failure.<\/em><\/p>\n<p>Maybe you add a dedicated backup network, and maybe you add a 2nd Live Migration network.<\/p>\n<p>How you get these connections is another story.\u00a0 Thanks to native NIC teaming, DCB, QoS, and a lot of other networking changes\/additions, there\u2019s lots of ways to get these 6+ communication paths in Windows Server 2012. \u00a0For that, you need to read about <a href=\"https:\/\/aidanfinn.com\/?p=12646\">converged fabrics<\/a>.<\/p>\n<div id=\"scid:0767317B-992E-4b12-91E0-4F059A8CECA8:eff1759d-5f66-44ef-b4f0-29be981eb341\" class=\"wlWriterEditableSmartContent\" style=\"margin: 0px; display: inline; float: none; padding: 0px;\">Technorati Tags: <a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Windows+Server+2012\">Windows Server 2012<\/a>,<a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Hyper-V\">Hyper-V<\/a>,<a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Storage\">Storage<\/a>,<a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Virtualisation\">Virtualisation<\/a>,<a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Networking\">Networking<\/a><\/div>\n","protected":false},"excerpt":{"rendered":"<p>This article was written just after the beta of WS2012 was launched.\u00a0 We now now that the performance of SMB 3.0 is -really- good, e.g. 1 million IOPS from a VM good. WS2012 is bringing a lot of changes in how we design storage for our Hyper-V hosts.\u00a0 There\u2019s no one right way, just lots &hellip; <a href=\"https:\/\/aidanfinn.com\/?p=12167\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Windows Server 2012 Hyper-V Storage Strategies&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[20],"tags":[181,80,99,195,118],"class_list":["post-12167","post","type-post","status-publish","format-standard","hentry","category-hyper-v","tag-hyper-v","tag-networking","tag-storage","tag-virtualisation","tag-windows-server-2012"],"aioseo_notices":[],"jetpack_featured_media_url":"","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/12167","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=12167"}],"version-history":[{"count":0,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/12167\/revisions"}],"wp:attachment":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=12167"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=12167"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=12167"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}