{"id":15421,"date":"2013-09-05T13:00:00","date_gmt":"2013-09-05T12:00:00","guid":{"rendered":"https:\/\/aidanfinn.com\/?p=15421"},"modified":"2013-09-05T13:00:00","modified_gmt":"2013-09-05T12:00:00","slug":"comparing-tcpip-compressed-and-smb-ws2012-r2-hyper-v-live-migration-speeds","status":"publish","type":"post","link":"https:\/\/aidanfinn.com\/?p=15421","title":{"rendered":"Comparing TCPIP, Compressed, and SMB WS2012 R2 Hyper-V Live Migration Speeds"},"content":{"rendered":"<p>I\u2019m building a demo for some upcoming events, blatantly ripping off what Ben Armstrong did at TechEd &#8211; copying is the best form of flattery, Ben <img decoding=\"async\" class=\"wlEmoticon wlEmoticon-smile\" style=\"border-top-style: none; border-left-style: none; border-bottom-style: none; border-right-style: none\" alt=\"Smile\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2013\/09\/wlEmoticon-smile.png\" \/>&#160; In the demo, I have 2 Dell R420 hosts with a bunch of NICs:<\/p>\n<ul>\n<li>2 disabled 1 GbE NICs <\/li>\n<li>2 Enabled 1 GbE NICs teamed for Live Migration <\/li>\n<li>2 10 GbE iWARP (RDMA) NICs not teamed for cluster, SMB Live Migration, and SMB 3.0 storage <\/li>\n<li>2 10 GbE NICs teamed for VM networking and host management <\/li>\n<\/ul>\n<p>It\u2019s absolutely over the top for real world but it gives me demo flexibility, especially to do the following.&#160; In the demo, I have a PowerShell script that will perform a measured Live Migration of a VM with 8 GB RAM (statically assigned).&#160; The VM is a pretty real workload: it\u2019s running WS2012 R2, SQL Server, and VMM 2012 R2.<\/p>\n<p>The script then does:<\/p>\n<ol>\n<li>Configure the cluster to use the 1 GbE team for Live Migration with TCPIP Live Migration <\/li>\n<li>Live migrate the VM (measured) <\/li>\n<li>Configure the cluster to use the 1 GbE team for Live Migration with Compressed Live Migration <\/li>\n<li>Live migrate the VM (measured) <\/li>\n<li>Configure the cluster to use a <em>single<\/em> 10 GbE iWARP NIC Live Migration with SMB Live Migration (SMB Direct) <\/li>\n<li>Live migrate the VM (measured) <\/li>\n<li>Configure the cluster to use a <em>both <\/em>10 GbE iWARP NIC Live Migration with SMB Live Migration (SMB Direct + Multichannel) <\/li>\n<li>Live migrate the VM (measured)\n<ol><!--EndFragment--><\/ol>\n<p> <!--EndFragment--><\/li>\n<\/ol>\n<p>What I observed in my test runs:<\/p>\n<ul>\n<li>TCP\/IP: About 95% of a 1 GbE NIC is utilised consistently for the duration. <\/li>\n<li>Compressed: The bandwidth utilisation has a saw tooth pattern up to around 98%, as one should expect with the dynamic nature of compression.&#160; CPU utilisation is higher (as expected), but remember that Live Migration will switch to TCP\/IP if compression is contending for resources with the host\/VMs. <\/li>\n<li>SMB Direct: <img decoding=\"async\" class=\"wlEmoticon wlEmoticon-smile\" style=\"border-top-style: none; border-left-style: none; border-bottom-style: none; border-right-style: none\" alt=\"Smile\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2013\/09\/wlEmoticon-smile.png\" \/> Nearly 10 Gbps over a single NIC. <\/li>\n<li>SMB Direct + SMB Multichannel: <img decoding=\"async\" class=\"wlEmoticon wlEmoticon-openmouthedsmile\" style=\"border-top-style: none; border-left-style: none; border-bottom-style: none; border-right-style: none\" alt=\"Open-mouthed smile\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2013\/09\/wlEmoticon-openmouthedsmile.png\" \/> Nearly 20 Gbps over the two iWARP rNICs. <\/li>\n<\/ul>\n<p>And the time taken for each Live Migration?<\/p>\n<p align=\"center\"><a href=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2013\/09\/image5.png\"><img loading=\"lazy\" decoding=\"async\" title=\"image\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"image\" src=\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2013\/09\/image_thumb5.png\" width=\"600\" height=\"305\" \/><\/a><\/p>\n<p>Over 78 seconds to move a running VM over a 1 GbE network without optimizations!&#160; Imagine that scaled out to a host with 250 GB RAM of production VM memory, needing to be drained for preventative maintenance.&#160; That\u2019s over 40 minutes, but it could be longer.&#160; That\u2019s a long time to wait to get critical services off of a host before a hardware warning becomes a host failure.<\/p>\n<p>As the Live Migrations get faster they get closer to the theoretical minimum time.&#160; There are four operations:<\/p>\n<ol>\n<li>Build the VM on the destination host (that magic 3% point, where the VM\u2019s dependencies are attempted to be prepared) <\/li>\n<li>Copy RAM <\/li>\n<li>Sync RAM if required <\/li>\n<li>Destroy the VM on the source host <\/li>\n<\/ol>\n<p>The first and last operation cannot be accelerated, generally taking a couple of seconds each.&#160; In fact, the first operation could take longer if you use Virtual Fiber Channel.&#160; <\/p>\n<p>This test with with a more common VM with 8 GB RAM.&#160; Remember that I <a href=\"https:\/\/aidanfinn.com\/?p=15291\" target=\"_blank\">moved a VM with 56 GB RAM in 35 seconds<\/a> using SMB Direct + Multichannel?&#160; That test was 33 seconds earlier today on the same preview release.&#160; Hmm, I think that hardware would take 2.5 minutes to drain 250 GM RAM of VMs, versus 42 minutes of un-optimised Live Migrations.&#160; I hope the point of this post is clear; if you need dense hosts then:<\/p>\n<ul>\n<li>Use 10 GbE Networking; If you can\u2019t upgrade to WS2012 R2 Hyper-V and use compression <\/li>\n<li>If you\u2019re using rNICs for storage then leverage that bandwidth and offload for optimising Live Migration, subject to QoS and SMB Bandwidth Constraints <\/li>\n<\/ul>\n<div id=\"scid:0767317B-992E-4b12-91E0-4F059A8CECA8:d8c48ff6-5356-45a1-b00f-e913c1eebc59\" class=\"wlWriterEditableSmartContent\" style=\"float: none; padding-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px\">Technorati Tags: <a href=\"http:\/\/technorati.com\/tags\/Windows+Server+2012+R2\" rel=\"tag\">Windows Server 2012 R2<\/a>,<a href=\"http:\/\/technorati.com\/tags\/Hyper-V\" rel=\"tag\">Hyper-V<\/a>,<a href=\"http:\/\/technorati.com\/tags\/Virtualisation\" rel=\"tag\">Virtualisation<\/a>,<a href=\"http:\/\/technorati.com\/tags\/Live+Migration\" rel=\"tag\">Live Migration<\/a>,<a href=\"http:\/\/technorati.com\/tags\/Networking\" rel=\"tag\">Networking<\/a><\/div>\n","protected":false},"excerpt":{"rendered":"<p>I\u2019m building a demo for some upcoming events, blatantly ripping off what Ben Armstrong did at TechEd &#8211; copying is the best form of flattery, Ben &#160; In the demo, I have 2 Dell R420 hosts with a bunch of NICs: 2 disabled 1 GbE NICs 2 Enabled 1 GbE NICs teamed for Live Migration &hellip; <a href=\"https:\/\/aidanfinn.com\/?p=15421\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Comparing TCPIP, Compressed, and SMB WS2012 R2 Hyper-V Live Migration Speeds&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[20],"tags":[181,74,80,195,120],"class_list":["post-15421","post","type-post","status-publish","format-standard","hentry","category-hyper-v","tag-hyper-v","tag-live-migration","tag-networking","tag-virtualisation","tag-windows-server-2012-r2"],"aioseo_notices":[],"jetpack_featured_media_url":"","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/15421","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=15421"}],"version-history":[{"count":0,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/15421\/revisions"}],"wp:attachment":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=15421"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=15421"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=15421"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}