![](\"https:\/\/aidanfinn.com\/wp-content\/uploads\/2012\/06\/image22.png\")
<\/p>\n
Why is this done? There are two reasons in WS2012:<\/p>\n
- \n
- There is a metadata operation taking place: this could be a VM start (very very quick), a dynamic virtual hard disk expansion, or even the creation of a fixed virtual hard disk.<\/li>\n
- A node loses direct connectivity to the storage and uses redirect IO to avoid VM storage corruption or VM outage.<\/li>\n<\/ul>\n
Note that redirected IO was used for Volume Shadow Copy Snapshot (VSS powered backup) operations in W2008 R2 but that is no longer the case in WS2012; it uses a new distributed snapshot mechanism to simply the backup.<\/p>\n
Why Did Redirected IO Kick In Here?<\/h4>\n
Restoring the VM was a backup operation, right? No; it\u2019s the complete opposite \u2013 it\u2019s a restore <\/em>operation and has absolutely nothing to do with VSS. Effectively the restore is creating files on a CSV \u2026 and that is one big mutha of a metadata operation. I wasn\u2019t there at the time and I don\u2019t know the backup tool, but I suspected that the admin restored the VM to a host that was not the owner of the target CSV. And that created tonnes of redirected IO:<\/p>\n
- \n
- The backup application restored some 500 GB of VM to the destination node<\/li>\n
- The node was not the CSV owner so redirected IO kicked in<\/li>\n
- 500 GB of data went redirecting from the \u201crestore\u201d host to the CSV owner across 1 GbE networking and then into the storage<\/li>\n
- Redirected IO finished up once the metadata operation (the restore) was completed<\/li>\n<\/ol>\n
That\u2019s my Dr. House theory anyway. I think I have it right \u2013 all the pieces fit. Of course, sometimes the patient died in House.<\/p>\n
Didn\u2019t They Control The Redirected IO Network?<\/h4>\n
They thought they did. The actually implemented the correct solution \u2026 for W2008 R2<\/a>. That solution manipulates (if necessary) the network metric value to make the CSV network the one with the lowest metric. This is a WS2012 R2 cluster and things are different.<\/p>\n
The clue why things are different is in the above illustration. There are two layers of redirected IO (the new lower block approach is 4 times faster) and bother are powered by SMB 3.0. What\u2019s one of the new features of SMB 3.0? SMB Multichannel. And what does SMB Multichannel do? It finds every connection between the SMB client (the non-CSV owner) and the SMB server (the CSV owner) and uses all those networks to stream data as quickly as possible.<\/p>\n
That\u2019s great for getting the SMB transfer done as quickly as possible, but it can be damaging (as I suspected it was here) if left unmanaged. And that\u2019s why we have features in WS2012 like QoS and SMB Constraints<\/a>.<\/p>\n
The Problem & Solution<\/h4>\n
In this case the hosts had a number of networks. The two that are relevant are the cluster network and the management network:<\/p>\n
- \n
- Cluster network: A private network with 2 * 1 GbE NICs in a team<\/li>\n
- Management network: 2 * 1 GbE in a team<\/li>\n<\/ul>\n
Two networks and both were eligible for SMB 3.0 to use for SMB Multichannel during Redirected IO. And that\u2019s what I suspect happened \u2026 500 GB of data went streaming across both NIC teams for the duration of the restore operation. That flooded the management network. That made it impossible for administrators to remotely manage or RDP into the hosts in the cluster.<\/p>\n
The solution in this case would be to use SMB constraints to control which networks would be used by SMB Multichannel<\/em> at the client end<\/em>. This cluster is simple enough in that regard. It uses iSCSI storage and SMB Multichannel should only be used by the team \u2026 so the team\u2019s interfaces would be listed in the cmdlet. Now SMB Multichannel will only use the team.<\/p>\n
I didn\u2019t implement the solution \u2013 that is outside my scope on the project \u2013 but I\u2019ve made the recommendation. I\u2019ll update if I get any news.<\/p>\n
In these types of clusters I\u2019d recommend this solution as a part of the build process. It gets a bit more complicated if you\u2019re using SMB 3.0 storage: just apply constraints to select any NIC that has a valid SMB Multichannel role, e.g. storage and redirected IO.<\/p>\n
Technorati Tags: Windows Server 2012<\/a>,Hyper-V<\/a>,Networking<\/a>,Failover Clustering<\/a>,Storage<\/a><\/div>\n","protected":false},"excerpt":{"rendered":"I was recently doing some System Center work on a customer site when they experienced a peculiar problem with their Windows Server 2012 (WS2012) Hyper-V cluster. They were recovering a very large virtual machine. OK \u2013 nothing weird there. What was weird was that they lost management access to the hosts as soon as the … Continue reading “Constraining SMB Powered Redirected IO Traffic To Selected Networks In A WS2012 Hyper-V Cluster”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"_wpcom_ai_launchpad_first_post":false,"footnotes":""},"categories":[20],"tags":[63,181,80,99,118],"class_list":["post-15221","post","type-post","status-publish","format-standard","hentry","category-hyper-v","tag-failover-clustering","tag-hyper-v","tag-networking","tag-storage","tag-windows-server-2012"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t