{"id":14547,"date":"2013-05-17T16:04:13","date_gmt":"2013-05-17T15:04:13","guid":{"rendered":"https:\/\/aidanfinn.com\/?p=14547"},"modified":"2013-05-17T16:04:13","modified_gmt":"2013-05-17T15:04:13","slug":"a-converged-networks-design-for-hyper-v-on-smb-3-0-storage-with-rdma-smb-direct","status":"publish","type":"post","link":"https:\/\/aidanfinn.com\/?p=14547","title":{"rendered":"A Converged Networks Design For Hyper-V On SMB 3.0 Storage With RDMA (SMB Direct)"},"content":{"rendered":"<p><em>When you are done reading this post, then see the update that I <a href=\"https:\/\/aidanfinn.com\/?p=14879\">added <\/a>for SMB Live Migration on Windows Server 2012 R2 Hyper-V.<\/em><\/p>\n<p>Unless you\u2019ve been hiding under a rock for the last 18 months, you might know that Windows Server 2012 (WS2012) Hyper-V (and IIS and SQL Server) supports storing content (such as virtual machines) on SMB 3.0 (WS2012) file servers (and scale-out file server active\/active clusters).\u00a0 The performance of this stuff goes from matching\/slightly beating iSCSI on 1 GbE, to crushing fiber channel on 10 GbE or faster.<\/p>\n<p>Big pieces of this design are SMB Multichannel (think simple, configuration free &amp; dynamic MPIO for SMB traffic) and SMB Direct (RDMA \u2013 low latency and CPU impact with non-TCP SMB 3.0 traffic).\u00a0 How does one network this design?\u00a0 RDMA is the driving force in the design.\u00a0 I\u2019ve talked to a lot of people about this topic over the last year. They normally over think the design, looking for solutions to problems that don\u2019t exist.\u00a0 In my core market, I don\u2019t expect lots of RDMA and Infiniband NICs to appear.\u00a0 But I thought I\u2019d post how I might do a network design.\u00a0 iWarp was in my head for this because I\u2019m hoping I can pitch the idea for my lab at the office. <img decoding=\"async\" class=\"wlEmoticon wlEmoticon-smile\" style=\"border-style: none;\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2013\/05\/wlEmoticon-smile.png\" alt=\"Smile\" \/><\/p>\n<p><a href=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2013\/05\/image2.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; float: none; padding-top: 0px; padding-left: 0px; margin-left: auto; display: block; padding-right: 0px; margin-right: auto; border: 0px;\" title=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2013\/05\/image_thumb2.png\" border=\"0\" alt=\"image\" width=\"604\" height=\"274\" \/><\/a><\/p>\n<p>On the left we have 1 or more Hyper-V hosts.\u00a0 There are up to 64 nodes in a cluster, and potentially lots of clusters connecting to a single SOFS \u2013 not necessarily 64 nodes in each!<\/p>\n<p>On the right, we have between 2 and 8 file servers that make up a Scale-Out File Server (SOFS) cluster with SAS attached (SAN or JBOD\/Storage Spaces) or Fiber Channel storage.\u00a0 More NICs would be required for iSCSI storage for the SOFS, probably using physical NICs with MPIO.<\/p>\n<p>There are 3 networks in the design:<\/p>\n<ul>\n<li>The Server\/VM networks.\u00a0 They might be flat, but in this kind of design I\u2019d expect to see some VLANs.\u00a0 Hyper-V Network Virtualization might be used for the VM Networks.<\/li>\n<li>Storage Network 1.\u00a0 This is one isolated and non-routed subnet, primarily for storage traffic.\u00a0 It will also be used for Live Migration and Cluster traffic.\u00a0 It\u2019s 10 GbE or faster and it\u2019s already isolated so it makes sense to me to use it.<\/li>\n<li>Storage Network 2.\u00a0 This is a second isolated and non-routed subnet.\u00a0 It serves the same function as Storage Network 2.<\/li>\n<\/ul>\n<p>Why 2 storage networks, ideally on 2 different switches?\u00a0 Two reasons:<\/p>\n<ul>\n<li>SMB Multichannel: It requires each multichannel NIC to be on a different subnet when connecting to a clustered file server, which includes the SOFS role.<\/li>\n<li>Reliable cluster communications: I have 2 networks for my cluster communications traffic, servicing my cluster design need for a reliable heartbeat.<\/li>\n<\/ul>\n<p>The NICs used for the SMB\/cluster traffic are NOT teamed.\u00a0 Teaming does not work with RDMA.\u00a0 Each physical rNIC has it\u2019s own IP address for the relevant (isolated and non-routed) storage subnet.\u00a0 These NICs do not go through the virtual switch so the easy per-vNIC QoS approach I\u2019ve mostly talked about is not applicable.\u00a0 Note that RDMA is not TCP.\u00a0 This means that when an SMB connection streams data, the OS packet scheduler cannot see it.\u00a0 That rules out OS Packet Scheduler QoS rules.\u00a0 Instead, you will need rNICs that support Datacenter Bridging (DCB) and your switches must also support DCB.\u00a0 You basically create QoS rules on a per-protocol-basis and push them down to the NICs to allow the hardware (which sees all traffic) to apply QoS and SLAs.\u00a0 This also has a side effect of less CPU utilization.<\/p>\n<p>Note: SMB traffic is restricted to the rNICs by using the constraint option.<\/p>\n<p>In the host(s), the management traffic does not go through the rNICs \u2013 they are isolated and non-routed.\u00a0 Instead, the Management OS traffic (monitoring, configuration, remote desktop, domain membership, etc) all goes through the virtual switch using a virtual NIC.\u00a0 Virtual NIC QoS rules are applied by the virtual switch.<\/p>\n<p>In the SOFS cluster nodes, management traffic will go through a traditional (WS2012) NIC team.\u00a0 You probably should apply per-protocol QoS rules on the management OS NIC for things like remote management, RDP, monitoring, etc.\u00a0 OS Packet Scheduler rules will do because you\u2019re not using RDMA on these NICs and this is the cheapest option.\u00a0 Using DCB rules here can be done but it requires end-to-end (NIC, switch, switch, etc, NIC) DCB support to work.<\/p>\n<p>What about backup traffic?\u00a0 I can see a number of options.\u00a0 Remember: with SMB 3.0 traffic, the agent on the hosts causes VSS to create a coordinated VSS snapshot, and the backup server retrieves backup traffic from a permission controlled (Backup Operators) hidden share on the file server or SOFS (yes, your backup server will need to understand this).<\/p>\n<ol>\n<li>Dual\/Triple Homed Backup Server: The backup server will be connected to the server\/VM networks.\u00a0 It will also be connected to one or both of the storage networks, depending on how much network resilience you need for backup, and what your backup product can do.\u00a0 A QoS (DCB) rule(s) will be needed for the backup protocol(s).<\/li>\n<li>A dedicated backup NIC (team): A single (or teamed) physical NIC (team) will be used for backup traffic on the host and SOFS nodes.\u00a0 No QoS rules are required for backup traffic because it is alone on the subnet.<\/li>\n<li>Create a backup traffic VLAN, trunk it through to a second vNIC (bound to the VLAN) in the hosts via the virtual switch.\u00a0 Apply QoS on this vNIC.\u00a0 In the case of the SOFS nodes, create a new team interface and bind it to the backup VLAN.\u00a0 Apply OS Packet Scheduler rules on the SOFS nodes for management and backup protocols.<\/li>\n<\/ol>\n<p>With this design you get all the connectivity, isolation, and network path fault tolerance that you might have needed with 8 NICs plus fiber channel\/SAS HBAs, but with superior storage performance.\u00a0 QoS is applied using DCB to guarantee minimum levels of service for the protocols over the rNICs.<\/p>\n<p>In reality, it\u2019s actually a simple design.\u00a0 I think people over think it, looking for a NIC team or protocol connection process for the rNICs.\u00a0 None of that is actually needed.\u00a0 You have 2 isolated networks, and SMB Multichannel figures it out for itself (it makes MPIO look silly, in my opinion).<\/p>\n<p>The networking chapter of Windows Server 2012 Hyper-V Installation And Configuration Guide goes from the basics through to the advanced steps of understanding these concepts and implementing them:<\/p>\n<ul>\n<li><a href=\"http:\/\/www.amazon.com\/Windows-Server-Hyper-V-Installation-Configuration\/dp\/1118486498%3FSubscriptionId%3DAKIAIJ5WNI7ZSH7W4OXA%26tag%3Dafm0c-20%26linkCode%3Dxm2%26camp%3D2025%26creative%3D165953%26creativeASIN%3D1118486498\">Amazon.com<\/a><\/li>\n<li><a href=\"http:\/\/www.amazon.co.uk\/gp\/product\/1118486498\/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&amp;camp=1634&amp;creative=6738&amp;creativeASIN=1118486498&amp;linkCode=as2&amp;tag=aidfin-21\">Amazon.co.uk<\/a><\/li>\n<li><a href=\"http:\/\/www.amazon.de\/gp\/product\/1118486498?ie=UTF8&amp;camp=1638&amp;creativeASIN=1118486498&amp;linkCode=xm2&amp;tag=aidfin0f-21\">Amazon.de<\/a><\/li>\n<\/ul>\n<div id=\"scid:0767317B-992E-4b12-91E0-4F059A8CECA8:5549323f-3cab-4b2f-afe1-1bef52498a48\" class=\"wlWriterEditableSmartContent\" style=\"float: none; margin: 0px; display: inline; padding: 0px;\">Technorati Tags: <a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Windows+Server+2012\">Windows Server 2012<\/a>,<a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Hyper-V\">Hyper-V<\/a>,<a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Virtualisation\">Virtualisation<\/a>,<a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Storage\">Storage<\/a>,<a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Failover+Clustering\">Failover Clustering<\/a><\/div>\n","protected":false},"excerpt":{"rendered":"<p>When you are done reading this post, then see the update that I added for SMB Live Migration on Windows Server 2012 R2 Hyper-V. Unless you\u2019ve been hiding under a rock for the last 18 months, you might know that Windows Server 2012 (WS2012) Hyper-V (and IIS and SQL Server) supports storing content (such as &hellip; <a href=\"https:\/\/aidanfinn.com\/?p=14547\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;A Converged Networks Design For Hyper-V On SMB 3.0 Storage With RDMA (SMB Direct)&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[20],"tags":[63,181,99,195,118],"class_list":["post-14547","post","type-post","status-publish","format-standard","hentry","category-hyper-v","tag-failover-clustering","tag-hyper-v","tag-storage","tag-virtualisation","tag-windows-server-2012"],"aioseo_notices":[],"jetpack_featured_media_url":"","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/14547","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=14547"}],"version-history":[{"count":0,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/14547\/revisions"}],"wp:attachment":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=14547"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=14547"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=14547"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}