{"id":15291,"date":"2013-08-09T17:02:49","date_gmt":"2013-08-09T16:02:49","guid":{"rendered":"https:\/\/aidanfinn.com\/?p=15291"},"modified":"2013-08-09T17:02:49","modified_gmt":"2013-08-09T16:02:49","slug":"testing-smb-live-migration-on-ws2012-r2-hyper-v","status":"publish","type":"post","link":"https:\/\/aidanfinn.com\/?p=15291","title":{"rendered":"Testing SMB Live Migration on WS2012 R2 Hyper-V"},"content":{"rendered":"<p>Today I got \u201cgeneration 2\u201d of the lab functioning the way I want it to today.&#160; The hosts are two Dell R420 12th generation servers, with 2 * 6 Core CPUs (24 logical processors each), 64 GB RAM and an extra Chelsio T440 CR quad port iWARP SFP+ NICs (for RDMA\/SMB Direct).&#160; The HP DL360 G7\u2019s are now the nodes in my Scale-Out File Server.<\/p>\n<p>2 of the iWARP NICs are used for the vSwitch NIC team.&#160; The other two are <em>not teamed<\/em> (prevents RDMA) and are on different subnets to support Multichannel to the SOFS.<\/p>\n<p>I have a script that tests the migration of a VM using the different WS2012 R2 options and times the movements.&#160; I just compared TCP\/IP Live Migration (over 1 * 10 GbE with some CPU impact) and compared it with SMB Live Migration which used 2 * 10 GbE.&#160; This was done with a single VM with 56 GB of statically assigned RAM.&#160; The results are in:<\/p>\n<ul>\n<li><font style=\"background-color: #0000ff\">TCP Live Migration: using around 9.8 Gbps took 58 seconds (which is excellent)<\/font><\/li>\n<li><font style=\"background-color: #0000ff\">SMB Live Migration: using nearly all of the available 20 Gbps took 35 seconds<\/font><\/li>\n<\/ul>\n<p>Think about that \u2026 a Linux (did I mention that?) VM with 56 GM RAM moved between two hosts in 35 seconds \u2026 with no noticeable CPU impact on the hosts caused by Live Migration!<\/p>\n<p>I actually moved 50 VMs concurrently yesterday and there was no noticeable CPU impact!<\/p>\n<p>There was a little engineering required:<\/p>\n<ul>\n<li>Jumbo Frames was configured on the NICs and (thanks to <a href=\"http:\/\/workinghardinit.wordpress.com\/\" target=\"_blank\">Didier Van Hoye<\/a>, aka <a href=\"https:\/\/twitter.com\/WorkingHardInIT\" target=\"_blank\">@workinghardinit<\/a>) I verified it end-to-end using ping &lt;IP&gt; \u2013l 8400 \u2013f.&#160; This gave me 10 Gbps on a single NIC.<\/li>\n<li>The final piece was to update the driver \u2026 the out of box driver refused to use more than 5 Gbps on each NIC via SMB Multichannel, usually sitting at 2.4 Gbps most of the time.&#160; Now I had 20 Gbps.<\/li>\n<li>I verified that RDMA was kicking in almost immediately via PerfMon.&#160; Multichannel is kicking in almost immediately too.<\/li>\n<\/ul>\n<div id=\"scid:0767317B-992E-4b12-91E0-4F059A8CECA8:6c563b67-184e-4006-a750-b4d05e34e083\" class=\"wlWriterEditableSmartContent\" style=\"float: none; padding-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px\">Technorati Tags: <a href=\"http:\/\/technorati.com\/tags\/Windows+Server+2012+R2\" rel=\"tag\">Windows Server 2012 R2<\/a>,<a href=\"http:\/\/technorati.com\/tags\/Hyper-V\" rel=\"tag\">Hyper-V<\/a>,<a href=\"http:\/\/technorati.com\/tags\/Virtualisation\" rel=\"tag\">Virtualisation<\/a>,<a href=\"http:\/\/technorati.com\/tags\/Networking\" rel=\"tag\">Networking<\/a><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Today I got \u201cgeneration 2\u201d of the lab functioning the way I want it to today.&#160; The hosts are two Dell R420 12th generation servers, with 2 * 6 Core CPUs (24 logical processors each), 64 GB RAM and an extra Chelsio T440 CR quad port iWARP SFP+ NICs (for RDMA\/SMB Direct).&#160; The HP DL360 &hellip; <a href=\"https:\/\/aidanfinn.com\/?p=15291\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Testing SMB Live Migration on WS2012 R2 Hyper-V&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[20],"tags":[181,80,195,120],"class_list":["post-15291","post","type-post","status-publish","format-standard","hentry","category-hyper-v","tag-hyper-v","tag-networking","tag-virtualisation","tag-windows-server-2012-r2"],"aioseo_notices":[],"jetpack_featured_media_url":"","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/15291","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=15291"}],"version-history":[{"count":0,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/15291\/revisions"}],"wp:attachment":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=15291"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=15291"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=15291"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}