{"id":12924,"date":"2012-06-19T12:22:25","date_gmt":"2012-06-19T11:22:25","guid":{"rendered":"https:\/\/aidanfinn.com\/?p=12924"},"modified":"2012-06-19T12:22:25","modified_gmt":"2012-06-19T11:22:25","slug":"windows-server-2012-nic-teaming-and-multichannel","status":"publish","type":"post","link":"https:\/\/aidanfinn.com\/?p=12924","title":{"rendered":"Windows Server 2012 NIC Teaming and Multichannel"},"content":{"rendered":"<p>Notes from TechEd NA 2012 <a href=\"http:\/\/channel9.msdn.com\/Events\/TechEd\/NorthAmerica\/2012\/WSV314\" target=\"_blank\">WSV314<\/a>:<\/p>\n<p><a href=\"http:\/\/channel9.msdn.com\/Events\/TechEd\/NorthAmerica\/2012\/WSV314\" target=\"_blank\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; margin-left: auto; border-top: 0px; margin-right: auto; border-right: 0px; padding-top: 0px\" title=\"image\" border=\"0\" alt=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image41.png\" width=\"404\" height=\"182\" \/><\/a><\/p>\n<p><strong><u>Terminology<\/u><\/strong><\/p>\n<ul>\n<li>It is a Team, not NIC bonding, etc.<\/li>\n<li>A team is made of Team Members<\/li>\n<li>Team Interfaces are the virtual NICs that can connect to a team and have IP stacks, etc.&#160; You can call them tNICs to differentiate them from vNICs in the Hyper-V world.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image42.png\"><u><\/u><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; margin-left: auto; border-top: 0px; margin-right: auto; border-right: 0px; padding-top: 0px\" title=\"image\" border=\"0\" alt=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image_thumb37.png\" width=\"404\" height=\"204\" \/><\/a><\/p>\n<p><strong><u>Team Connection Modes<\/u><\/strong><\/p>\n<p>Most people don\u2019t know the teaming mode they select when using OEM products.&#160; MSFT are clear about what teaming does under the cover.&#160; Connection mode = how do you connect to the switch?<\/p>\n<ul>\n<li>Switch Independent can be used where the switch doesn\u2019t need to know anything about the team.<\/li>\n<li>Switch dependent teaming is when the switch does need to know something about the team. The switch decides where to send the inbound traffic.<\/li>\n<\/ul>\n<p>There are 2 switch dependent modes:<\/p>\n<ul>\n<li>LACP (Link Aggregation Control Protocol) is where the is where the host and switch agree on who the team members are. IEEE 802.1ax<\/li>\n<li>Static Teaming is where you configure it on the switch.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image43.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; margin-left: auto; border-top: 0px; margin-right: auto; border-right: 0px; padding-top: 0px\" title=\"image\" border=\"0\" alt=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image_thumb38.png\" width=\"404\" height=\"224\" \/><\/a><\/p>\n<p><strong><u>Load Distribution Modes<\/u><\/strong><\/p>\n<p>You also need to know how you will spread traffic across the team members in the team.<\/p>\n<p>1) <strong><u>Address Hash<\/u><\/strong> comes in 3 flavours:<\/p>\n<ul>\n<li>4-tuple (the default): Uses RSS on the TCP\/UDP ports.&#160; <\/li>\n<li>2-tuple: If the ports aren\u2019t available (encrypted traffic such as IPsec) then it\u2019ll go to 2-tuple where it uses the IP address.<\/li>\n<li>MAC address hash: If not IP traffic, then MAC addresses are hashed.<\/li>\n<\/ul>\n<p>2) We also have <strong><u>Hyper-V Port<\/u><\/strong>, where it hashes the port number on the Hyper-V switch that the traffic is coming from.&#160; Normally this equates to per-VM traffic.&#160; No distribution of traffic.&#160; It maps a VM to a single NIC.&#160; If a VM needs more pipe than a single NIC can handle then this won\u2019t be able to do it.&#160; Shouldn\u2019t be a problem because we are consolidating after all.<\/p>\n<p>Maybe create a team in the VM?&#160; Make sure the vNICs are on different Hyper-V Switches.&#160; <\/p>\n<p><strong><u>SR-IOV<\/u><\/strong><\/p>\n<p>Remember that SR-IOV bypasses the host stack and therefore can\u2019t be teamed at the host level.&#160; The VM bypasses it.&#160; You can team two SR-IOV enabled vNICs in the guest OS for LBFO.<\/p>\n<p><strong><u>Switch Independent \u2013 Address Hash<\/u><\/strong><\/p>\n<p>Outbound traffic in Address Hashing will spread across NICs. All inbound traffic is targeted at a single inbound MAC address for routing purposes, and therefore only uses 1 NIC.&#160; Best used when:<\/p>\n<ul>\n<li>Switch diversity is a concern<\/li>\n<li>Active\/Standby mode<\/li>\n<li>Heavy outbound but light inbound workloads<\/li>\n<\/ul>\n<p><strong><u>Switch Independent &#8211; Hyper-V Port<\/u><\/strong><\/p>\n<p>All traffic from each VM is sent out on that VM\u2019s physical NIC or team member.&#160; Inbound traffic also comes in on the same team member.&#160; So we can maximise NIC bandwidth.&#160; It also allows for maximum use of VMQs for better virtual networking performance.<\/p>\n<p>Best for:<\/p>\n<ul>\n<li>Number of VMs well exceeds number of team members<\/li>\n<li>You\u2019re OK with VM being restricted to bandwidth of a single team member<\/li>\n<\/ul>\n<p><strong><u>Switch Dependent Address Hash<\/u><\/strong><\/p>\n<p>Sends on all active members by using one of the hashing methods.&#160; Receives on all ports \u2013 the switch distributes inbound traffic.&#160; No association between inbound and outbound team members.&#160; Best used for:<\/p>\n<ul>\n<li>Native teaming for maximum performance and switch diversity is not required.<\/li>\n<li>Teaming under the Hyper-V switch when a VM needs to exceed the bandwidth limits of a single team member&#160; Not as efficient with VMQ because we can\u2019t predict the traffic.<\/li>\n<\/ul>\n<p>Best performance for both inbound and outbound.<\/p>\n<p><strong><u>Switch Dependent \u2013 Hyper-V Port<\/u><\/strong><\/p>\n<p>Sends on all active members using the hashed port \u2013 1 team member per VM.&#160; Inbound traffic is distributed by the switch&#160; on all ports so there is no correlation to inbound and outbound.&#160; Best used when:<\/p>\n<ul>\n<li>When number of VMs on the switch well exceeds the number of team members <strong><em><u>AND<\/u><\/em><\/strong><\/li>\n<li>You have a <em>policy<\/em> that says you must use switch dependent teaming.<\/li>\n<\/ul>\n<p>When using Hyper-V you will normally want to use Switch Independent &amp; Hyper-V Port mode.&#160; <\/p>\n<p>When using native physical servers you\u2019ll likely want to use Switch Independent &amp; Address Hash.&#160; Unless you have a policy that can\u2019t tolerate a switch failure.<\/p>\n<p><strong><u>Team Interfaces<\/u><\/strong><\/p>\n<p>There are different ways of interfacing with the team:<\/p>\n<ul>\n<li>Default mode: all traffic from all VLANs is passed through the team<\/li>\n<li>VLAN mode: Any traffic that matches a VLAN ID\/tag is passed through.&#160; Everything else is dropped.<\/li>\n<\/ul>\n<p>Inbound traffic passes through to one team interface at once.<\/p>\n<p><a href=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image44.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; margin-left: auto; border-top: 0px; margin-right: auto; border-right: 0px; padding-top: 0px\" title=\"image\" border=\"0\" alt=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image_thumb39.png\" width=\"404\" height=\"112\" \/><\/a><\/p>\n<p>The only supported configuration for Hyper-V is shown above: Default mode passing through all traffic t the Hyper-V Switch.&#160; Do all the VLAN tagging and filtering on the Hyper-V Switch.&#160; You cannot mix other interfaces with this team \u2013 the team must be dedicated to the Hyper-V Switch.&#160; REPEAT: This is the only supported configuration for Hyper-V.<\/p>\n<p>A new team has one team interface by default.&#160; <\/p>\n<p>Any team interfaces created <em>after<\/em> the initial team creation must be VLAN mode team interfaces (bound to a VLAN ID).&#160; You can delete these team interfaces.<\/p>\n<p>Get-NetAdapter: Get the properties of a team interface<\/p>\n<p>Rename-NetAdapter: rename a team interface<\/p>\n<p><strong><u>Team Members<\/u><\/strong><\/p>\n<ul>\n<li>Any physical ETHERNET adapter with a Windows Logo (for stability reasons and promiscuous mode for VLAN trunking) can be a team member.<\/li>\n<li>Teaming of InfiniBand, Wifi, WWAN not supported.<\/li>\n<li>Teams made up of teams not supported.<\/li>\n<\/ul>\n<p>You can have team members in active or standby mode.<\/p>\n<p><strong><u>Virtual Teams<\/u><\/strong><\/p>\n<p>Supported <em>if:<\/em><\/p>\n<ul>\n<li>No more than 2 team members in the guest OS team<\/li>\n<\/ul>\n<p>Notes:<\/p>\n<ul>\n<li>Intended for SR-IOV NICs but will work without it.<\/li>\n<li>Both vNICs in the team should be connected to different virtual switches on different physical NICs<\/li>\n<\/ul>\n<p>If you try to team a vNIC that is not on an External switch, it will show up fine and OK until you try to team it.&#160; Teaming will shut down the vNIC at that point.&#160; <\/p>\n<p>You also have to allow teaming in a vNIC in Advanced Properties \u2013 Allow NIC teaming.&#160; Do this for each of the VM\u2019s vNICs.&#160; Without this, failover will not succeed.&#160; <\/p>\n<p><strong><u>PowerShell CMDLETs for Teaming<\/u><\/strong><\/p>\n<p>The UI is actually using POSH under the hood.&#160; You can use the NIC Teaming UI to remotely manage\/configure a server using RSAT for Windows 8.&#160; WARNING: Your remote access will need to run over a NIC that you aren\u2019t altering because you would lose connectivity.<\/p>\n<p><a href=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image45.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; margin-left: auto; border-top: 0px; margin-right: auto; border-right: 0px; padding-top: 0px\" title=\"image\" border=\"0\" alt=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image_thumb40.png\" width=\"404\" height=\"84\" \/><\/a><\/p>\n<p><strong><u>Supported Networking Features<\/u><\/strong><\/p>\n<p>NIC teaming works with almost everything:<\/p>\n<p><a href=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image46.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; margin-left: auto; border-top: 0px; margin-right: auto; border-right: 0px; padding-top: 0px\" title=\"image\" border=\"0\" alt=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image_thumb41.png\" width=\"404\" height=\"172\" \/><\/a><\/p>\n<p>TCP Chimney Offload, RDMA and SR-IOV bypass the stack so obviously they cannot be teamed in the host.<\/p>\n<p><strong><u>Limits<\/u><\/strong><\/p>\n<ul>\n<li>32 NICs in a team<\/li>\n<li>32 teams<\/li>\n<li>32 team interfaces in a team<\/li>\n<\/ul>\n<p>That\u2019s a lot of quad port NICs.&#160; Good luck with that! <img decoding=\"async\" style=\"border-bottom-style: none; border-left-style: none; border-top-style: none; border-right-style: none\" class=\"wlEmoticon wlEmoticon-winkingsmile\" alt=\"Winking smile\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/wlEmoticon-winkingsmile1.png\" \/>&#160;<\/p>\n<p><strong><u>SMB Multichannel<\/u><\/strong><\/p>\n<p>An alternative to a team in an SMB 3.0 scenario.&#160; Can use multiple NICs with same connectivity, and use multiple cores via NIC RSS to have simultaneous streams over a single NIC (RSS) or many NICs (teamed, not teamed, and also with RSS if available).&#160; Basically, leverage more bandwidth to get faster SMB 3.0 throughput.<\/p>\n<p>Without it, a 10 GbE NIC would only be partly used by SMB \u2013 single CPU core trying to transmit.&#160; RSS makes it multi-threaded\/core, and therefore many connections by the data transfer.<\/p>\n<p>Remember \u2013 you cannot team RDMA.&#160; So another case to use Multichannel and get an LBFO effect is to use SMB Multichannel \u2026. or I should say \u201cuse\u201d \u2026 SMB 3.0 turns it on automatically if multiple paths are available between client and server.<\/p>\n<p>SMB 3.0 is NUMA aware.<\/p>\n<p>Multichannel will only use NICs of same speed\/type.&#160; Won\u2019t see traffic spread over a 10 GbE and a 1 GbE NIC, for example, or over RDMA-enabled and non-RDMA NICs.&#160; <\/p>\n<p>In tests, the throughput on RSS enabled 10 GbE NICs (1, 2, 3, and 4 NICs), seemed to grow in a predictable near-linear rate.<\/p>\n<p>SMB 3.0 uses a shortest queue first algorithm for load balancing \u2013 basic but efficient.<\/p>\n<p><strong><u>SMB Multichannel and Teaming<\/u><\/strong><\/p>\n<p>Teaming allows for faster failover.&#160; MSFT recommending teaming where applicable.&#160; Address-hash port mode with Multichannel can be a nice solution.&#160; Multichannel will detect a team and create multiple connections over the team.<\/p>\n<p><strong><u>RDMA<\/u><\/strong><\/p>\n<p>If RDMA is possible on both client and server then SMB 3.0 switches over to SMB Direct.&#160; Net monitoring will see negotiation, and then \u2026 \u201csilence\u201d for the data transmission.&#160; Multichannel is supported across single or multiple NICs \u2013 no NIC teaming, remember!<\/p>\n<p><strong><u>Won\u2019t Work With Multichannel<\/u><\/strong><\/p>\n<ul>\n<li>Single non-RSS capable NIC<\/li>\n<li>Different type\/speed NICs, e.g. 10 GbE RDMA favoured over 10 GbE non-RDMA NIC<\/li>\n<li>Wireless can be failed from but won\u2019t be used in multi-channel<\/li>\n<\/ul>\n<p><strong><u>Supported Configurations<\/u><\/strong><\/p>\n<p>Note that Multichannel over a team of NICs is favoured over multichannel over the same NICs that are not in a team.&#160; Added benefits of teaming (types, and fast failover detection).&#160; This applies, whether the NICs are RSS capable or not.&#160; And the team also benefits non-SMB 3.0 traffic.<\/p>\n<p><a href=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image47.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; margin-left: auto; border-top: 0px; margin-right: auto; border-right: 0px; padding-top: 0px\" title=\"image\" border=\"0\" alt=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image_thumb42.png\" width=\"404\" height=\"229\" \/><\/a><\/p>\n<p><strong><u>Troubleshooting SMB Multichannel<\/u><\/strong><\/p>\n<p><a href=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image48.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; margin-left: auto; border-top: 0px; margin-right: auto; border-right: 0px; padding-top: 0px\" title=\"image\" border=\"0\" alt=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image_thumb43.png\" width=\"404\" height=\"222\" \/><\/a><\/p>\n<p>Plenty to think about there, folks!&#160; Where it applies in Hyper-V?<\/p>\n<ul>\n<li>NIC teaming obviously applies.<\/li>\n<li>Multichannel applies in the cluster: redirected IO over the cluster communications network<\/li>\n<li>Storing VMs on SMB 3.0 file shares<\/li>\n<\/ul>\n<div style=\"padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px\" id=\"scid:0767317B-992E-4b12-91E0-4F059A8CECA8:fc9cedb6-f6dd-48f1-85c2-b574792adf77\" class=\"wlWriterEditableSmartContent\">Technorati Tags: <a href=\"http:\/\/technorati.com\/tags\/Event+Notes\" rel=\"tag\">Event Notes<\/a>,<a href=\"http:\/\/technorati.com\/tags\/Windows+Server+2012\" rel=\"tag\">Windows Server 2012<\/a>,<a href=\"http:\/\/technorati.com\/tags\/Networking\" rel=\"tag\">Networking<\/a><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Notes from TechEd NA 2012 WSV314: Terminology It is a Team, not NIC bonding, etc. A team is made of Team Members Team Interfaces are the virtual NICs that can connect to a team and have IP stacks, etc.&#160; You can call them tNICs to differentiate them from vNICs in the Hyper-V world. Team Connection &hellip; <a href=\"https:\/\/aidanfinn.com\/?p=12924\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Windows Server 2012 NIC Teaming and Multichannel&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[14],"tags":[176,80,118],"class_list":["post-12924","post","type-post","status-publish","format-standard","hentry","category-eventnotes","tag-eventnotes","tag-networking","tag-windows-server-2012"],"aioseo_notices":[],"jetpack_featured_media_url":"","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/12924","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=12924"}],"version-history":[{"count":0,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/12924\/revisions"}],"wp:attachment":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=12924"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=12924"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=12924"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}