{"id":9566,"date":"2009-03-31T09:51:00","date_gmt":"1999-11-29T20:00:00","guid":{"rendered":"https:\/\/aidanfinn.com\/?p=9566"},"modified":"2009-03-31T09:51:00","modified_gmt":"1999-11-29T20:00:00","slug":"slow-network-bandwidth-vs-latency","status":"publish","type":"post","link":"https:\/\/aidanfinn.com\/?p=9566","title":{"rendered":"Slow Network: Bandwidth VS Latency"},"content":{"rendered":"<p>I\u2019ve done a little speaking about this subject over the last while and after listening to a recent radio conversation I thought I\u2019d post something too.\u00a0 The story was about a revolutionary online gaming system, the idea being that instead of buying a DVD with the game, you\u2019d play it online and it would stream to your PC or console.\u00a0 One of the expert commentators finally made the point I was waiting to hear.\u00a0 The service provider has their servers in the USA.\u00a0 This means that the games players in North America would be close to the servers.\u00a0 Until there was a European presence, games players here should probably steer clear because the interaction would be slow.\u00a0\n<\/p>\n<p>Why not just get a \u201cfaster\u201d Internet connection?\u00a0 That\u2019s our usual answer to these problems.\u00a0 Think about a business that has a head office with a SharePoint server and a branch office where users use that server over the WAN.\u00a0 When enough users call into the helpdesk to complain about slow downloads, what do we do?\u00a0 We usually go out and buy a bigger WAN link.\u00a0 That is the wrong thing to do without considering what\u2019s really going on.\n<\/p>\n<p><strong><u>Definitions<\/u><\/strong>\n<\/p>\n<p>There\u2019s two things to measure when it comes to a network link.<\/p>\n<ul>\n<li>Bandwidth: This is how much data we can transfer at once, i.e. in one packet, from the source to the destination.\u00a0 We measure this in KBPS, MBPS or even GBPS if you have lots of money.\n<\/li>\n<li>Latency: This is how long a packet takes to travel from the source to the destination.\u00a0 This is limited by the laws of physics.\u00a0 An electron can only travel at 1 speed on a copper wire.\u00a0 Increasing bandwidth has no affect on this.\u00a0 A photon on a fibre link can only travel at the speed of light.\u00a0 Even if we could increase this speed, Einstein tells us that this would negatively affect the latency!\u00a0 Finally, devices that process the data at various hops, e.g. routers, switches and firewalls, also add to latency.\u00a0 So, the further apart a client and server are, the longer a transmission takes.\u00a0 Adding in more network devices, e.g. transmissions between different ISP\u2019s, worsens this.<\/li>\n<\/ul>\n<p>There is a conversation between the server and the client when any data or a file is transferred over the network.\u00a0 The file is broken up into packets.\u00a0 Headers and control flags wrap each of those packets up to increase the amount of data.\u00a0 Then a conversation takes place.\u00a0 At a very high level, here\u2019s how it goes:<\/p>\n<ul>\n<li>Server: Here\u2019s packet 1\n<\/li>\n<li>Client: Acknowledged\n<\/li>\n<li>Server: Here\u2019s packet 2\n<\/li>\n<li>Client: Acknowledged\n<\/li>\n<li>Server: Here\u2019s packet 3\n<\/li>\n<li>Client: Not Acknowledged\n<\/li>\n<li>Server: Here\u2019s packet 3 again\n<\/li>\n<li>Client: Acknowledged\n<\/li>\n<li>Server: Here\u2019s packet 4\n<\/li>\n<li>Client: Acknowledged\n<\/li>\n<li>Server: Here\u2019s packet 5\n<\/li>\n<li>Client: Acknowledged\n<\/li>\n<li>Server: Here\u2019s packet 6\n<\/li>\n<li>Client: Acknowledged<\/li>\n<\/ul>\n<p>And so it goes until all of the packets that make up the entire file are transmitted.\u00a0 Bandwidth affects the time for transmission by reducing how much data we can put into a packet.\u00a0 Note that the TCP stack in the operating system can also limit this.\u00a0 Bandwidth also causes problems when we try to put too many simultaneous conversations onto a pipe.\u00a0 We can monitor bandwidth by measuring link utilisation.\n<\/p>\n<p>Latency is best explained as follows.\u00a0 If it takes 1 millisecond to transmit a packet between the client and server then the above file copy would take 14 milliseconds.\u00a0 If we move the client to a remove location then latency goes up, perhaps to 100 milliseconds.\u00a0 Now the file copy takes 100 times longer: 1400 milliseconds.\u00a0 Realistically, a file transfer requires exponentially more packets.\u00a0 An intercontinental latency measurement (use PING) might be 300 milliseconds or more!\n<\/p>\n<p>Let\u2019s go back to the above examples and see how latency and bandwidth affected them:<\/p>\n<ul>\n<li>The European Online games player: Games playing requires timely interaction with the game.\u00a0 The slower the game player\u2019s responses, the worse they play.\u00a0 If a European players responses are 30 times longer than those of an American, how can they have the same gaming experience?\u00a0 The European is hit by bandwidth.\n<\/li>\n<li>By throwing bandwidth at the SharePoint server, we allow many more users in the branch office to have the same slow experience.\u00a0 Latency causes the packets that make up the file transmission to be slow.<\/li>\n<\/ul>\n<p><strong><u>Basic Networking Solutions<\/u><\/strong>\n<\/p>\n<p>Networking wise, there\u2019s a few solutions we can look at:<\/p>\n<ul>\n<li>Place the servers closer to the clients.\u00a0 For a \u201ccloud computing\u201d service provider, that\u2019s possible by having the service closer to the consumers.\u00a0 For a corporate, that might mean having servers in the branch office, something we want to steer clear of doing if at all possible to reduce costs and complexity.\n<\/li>\n<li>Reduce the hops between the servers and clients: For a company, this means strategically subscribing to ISP services.\u00a0 For an online service provider you can only do this so much, e.g. use a major service provider.\u00a0 But there\u2019s always going to be clients who many hops away.<\/li>\n<\/ul>\n<p>We still have issues here.\u00a0 So we want to get more data on the pipe and once and send fewer packets so that latency plays less of a role.\n<\/p>\n<p><strong><u>Advanced Networking Solutions<\/u><\/strong>\n<\/p>\n<p>Both Riverbed (Steelhead) and Citrix (WanScaler) have appliances that can be placed in both the head office and the branch office.\u00a0 A PC in the branch office will look to copy a file from the HQ server.\u00a0 All the usual file locking and security stuff takes place (as it will throughout this process).\u00a0 The server breaks the file up into packets and starts the transfer.\u00a0 The appliance in the HQ sits silently between the server and the WAN connection.\u00a0 It listens to the stream and uses a hashing algorithm to break down the data transmission into blocks which are stored on the appliance according to a set of predefined rules.\u00a0\n<\/p>\n<p>The data travels over the WAN to the branch office.\u00a0 The branch office appliance also listens to the new data and does and identical breakdown and hashing algorithm ID of the blocks before caching them.\u00a0 The data stream continues to the client.\u00a0 At this point, no speed increase has taken place.\n<\/p>\n<p>The process will go as follows if this client or any other in the same branch office goes to transfer this file again.\u00a0 The second client does the usual file lock and security stuff.\u00a0 The server believes it is talking to the client.\u00a0 Instead it talks to the HQ appliance.\u00a0 The appliance breaks down the blocks and ID\u2019s them using the hashing algorithm.\u00a0 Any previously cached packets don\u2019t need to be transmitted.\u00a0 Instead, the HQ appliance works with the branch office appliance.\u00a0 The branch office appliance receives the block\u2019s hash ID and then sends that packet to the client over the LAN.\u00a0\n<\/p>\n<p>The effect?\u00a0 Previously transmitted blocks are not sent over the WAN.\u00a0 This reduces bandwidth utilisation.\u00a0 By removing the need to send data at all, we remove latency from the equation.\u00a0 Other than some security and file locking procedures, a file transfer can be local only at the branch office, i.e. between the appliance and the client.\n<\/p>\n<p>Because the system works by using blocks the optimisation can even work for files that haven\u2019t even been requested over the WAN before, as long as they are made up of blocks similar to previously transmitted files.\n<\/p>\n<p><em>The process I\u2019ve talked about here has been simplified.<\/em>\n<\/p>\n<p>The appliances work at a TCP level.\u00a0 This means that WAN optimisation can improve way more than just file copies, e.g. Exchange, Oracle, SQL, Lotus Notes, etc.\u00a0 The basic requirements are that the data is not signed and not encrypted.\u00a0 You also need to turn off SMB data signing in Group Policy.\u00a0 That\u2019s because the appliances are in-a-way performing a man-in-the-middle attack.\n<\/p>\n<p>These appliances are very expensive so they are not widespread.\u00a0 I\u2019ve done some work with low spec devices from Riverbed back in 2006 and they really did work very well.\n<\/p>\n<p><strong><u>The Next Generation TCP Stack<\/u><\/strong>\n<\/p>\n<p>Microsoft included a new TCP stack in Windows Vista and Windows Server 2008.\u00a0 It is also in Windows 7 and Windows Server 2008 R2.\u00a0 The Next Generation TCP Stack isn\u2019t the complete WAN solution but it does improve things.\n<\/p>\n<p>Compound TCP aims to reduce the effect of latency.\u00a0 The server in the previous example of the file transfer will send many packets before waiting for an acknowledgement:<\/p>\n<ul>\n<li>Server: Here\u2019s packet 1\n<\/li>\n<li>Server: Here\u2019s packet 2\n<\/li>\n<li>Server: Here\u2019s packet 3\n<\/li>\n<li>Server: Here\u2019s packet 4\n<\/li>\n<li>Server: Here\u2019s packet 5\n<\/li>\n<li>Server: Here\u2019s packet 6\n<\/li>\n<li>Client: Give me packet 3 again\n<\/li>\n<li>Server: Here\u2019s packet 3\n<\/li>\n<li>Client: Acknowledged<\/li>\n<\/ul>\n<p>As I said before, this is a tiny example of what is going on under the covers.\u00a0 If our latency was 100 milliseconds before then the first example took 1400 milliseconds.\u00a0 With compound TCP, the copy will take 900 milliseconds.\n<\/p>\n<p>Vista and Windows Server 2008 also gave us an Auto-Scaling Receive Side Window.\u00a0 The client and server work together to calculate how much bandwidth there is, i.e. how big a packet can be or how much data can be placed in the pipe at one time.\u00a0 In legacy operating systems such as XP and Windows Server 2003, this is a static definition for both LAN and WAN transfers and usually shouldn\u2019t be manually altered.\u00a0 With this auto scaling receive side window, our file copy will increase the size of the data portion of the packets and may look like this:<\/p>\n<li>Server: Here\u2019s packet 1\n<li>Server: Here\u2019s packet 2\n<li>Server: Here\u2019s packet 3\n<li>Client: Give me packet 3 again\n<li>Server: Here\u2019s packet 3\n<li>Client: Acknowledged\n<p>\u00a0\n<\/p>\n<p>We\u2019re using Compound TCP as well, meaning we\u2019re sending fewer packets <em>and<\/em> using as much bandwidth as possible by sending more at once.\u00a0 Now our time to transfer the file on the 100 millisecond link is 600 milliseconds.\u00a0 Remember this started out at 1400 milliseconds.\n<\/p>\n<p>The limits to the optimisation offered by Auto Scaling Receive Side Windows are (a) the ability for the application protocol to buffer data and (b) the bandwidth available.\n<\/p>\n<p><em>Microsoft came up with SMBv2 so that the file and print sharing protocol could handle this huge data streams that can now be transferred over large links.\u00a0 <\/em>\n<\/p>\n<p>The risk with this auto scaling receive side window is that one file copy over the WAN could shut consume the entire WAN link and effectively shut down business traffic like RDP, ICA, etc.\u00a0 Using Group Policy (GPO), we can tag traffic between selected sources, selected destinations, certain protocols (TCP or UDP) or ports (80, 443, 3389, etc).\u00a0 An example might be that all web traffic between 10.0.0.1\/24 and 10.195.34.0\/24 on TCP 80 should be tagged.\u00a0 Network administrators can then use those tags to put QoS (Quality of Service) rules in place for traffic prioritisation, e.g. TCP 80 traffic with the Internet or a proxy server might be of a lower priority than HTTP traffic with a SharePoint Server and RDP traffic with a Terminal Server might be higher again.\u00a0 This would prioritise critical business traffic over lesser valued traffic at the network level.\n<\/p>\n<p>This does improve things but data still has to go over the WAN and latency is still going to cause noticeable delays.\n<\/p>\n<p><em>Note that this huge improvement of data transmission really is best seen on dedicated local are networks between servers, e.g. application servers and data servers.<\/em>\n<\/p>\n<p><strong><u>Windows 7 and Window Server 2008 R2 \u2013 Better Together<\/u><\/strong>\n<\/p>\n<p>Microsoft is introducing BranchCache in Windows 7 (Enterprise and Ultimate editions only at the time of writing) and Windows Server 2008 R2.\u00a0 This will allow a Windows 7 client to access a branch office cache of whole files that are stored on a Windows Server 2008 R2 content server.\u00a0 The protocols being optimised are SMB (file sharing), HTTP and HTTPS (and logically, BITS).\u00a0 There are 2 architectures:<\/p>\n<ul>\n<li>Distributed: Clients in the branch office have a peer-to-peer network of sorts.\u00a0 A client starts to download a file from the HQ web or file server.\u00a0 The usual security stuff is done (as throughout this process).\u00a0 The client broadcasts on the LAN to see if other clients have the file cached.\u00a0 It uses a hash ID for the file that is obtained from the content server.\u00a0 If no other clients have it, the file is downloaded from the HQ.\u00a0 Another client now starts to download the file after the usual security stuff.\u00a0 It again broadcasts using the hash ID.\u00a0 The first client responds and the second client transfers the file over the LAN \u2013 not the WAN.\u00a0 This uses a broadcast model and is thus limited to a broadcast domain or single VLAN.\u00a0 There\u2019s also the problem that a PC might hibernate or be turned off, thus disabling it\u2019s cache content in the branch office.\n<\/li>\n<li>Hosted: A Windows Server 2008 R2 server is placed in the branch office.\u00a0 The branch office clients use unicasts to communicate with it rather than using the broadcast based peer-to-peer model.\u00a0 This tidies up the network, allows to multiple VLAN\u2019s and the cache is always on.<\/li>\n<\/ul>\n<p>The initial version of BranchCache only supports file based, not block level, caching.\u00a0 It also only caches the download.\u00a0 Uploads (saves) must be transferred over the WAN to the central server and are not optimised.\n<\/p>\n<p>All the settings of BranchCache are controllable using GPO.\u00a0 Content administrators can control it at the share and site level.\u00a0 Caches are secure and users can only access what the content share permissions allow for.\n<\/p>\n<p><strong><u>Move The User Interface<\/u><\/strong>\n<\/p>\n<p>We&#8217;re used to having the client (PC) in the branch office or out roaming on the Internet and the server in the data centre.\u00a0 We&#8217;ll always have some kind of latency when data has to travel between the central server and the remote client even if we use any of the above advanced solutions.\u00a0 What if the user &quot;logged in&quot; using a client that was close to the servers.\u00a0 Maybe that central client would be accessible from anywhere, no matter where the client was, e.g. in a branch office, hotel or at home.\u00a0 Terminal Services is a mature way of doing this.\u00a0 Citrix has built upon it for companies with larger TerSvcs server farm requirements.\u00a0 With these products, the user logs in using physical equipment but their session runs in the central data centre.\u00a0 Data travels only over the WAN, not over the LAN.\n<\/p>\n<p>Windows Server 2008 Terminal Services solved the biggest problem with this type of solution: printers.\u00a0 Terminal Services administrators were sick of printer driver issues on the servers.\u00a0 Thanks to EasyPrint you don&#8217;t have to deal with drivers any more &#8211; if the clients are running Windows 7, Vista SP1 or XP SP3.\u00a0 And users don&#8217;t have to wait half an hour for the print job to download.\u00a0 It&#8217;s near instant thanks to Microsoft&#8217;s XPS technology.\u00a0 Microsoft also added application publication, a SSL interface and the ability to securely access those from anywhere using the TS Gateway.\n<\/p>\n<p>Windows Server 2008 R2 rebrands this as Remote Desktop Services.\u00a0 This is beacause they&#8217;re adding a VDI broker to access virtualised desktops running on a central Hyper-V (machine virtualisation) farm.\u00a0 At the time of writing this is still a beta.\u00a0 You can access RTM solutions from Provision Networks and Citrix.\u00a0 I like the look of the Citrix solution because it looks pretty complete.\u00a0 The idea of VDI is that users access a familiar desktop environment, existing adminsitrative systems can be reused, application issues are minimal (Terminal Services can require &quot;application silos&quot; of application specific servers) and Helpdesk doesn&#8217;t need to do change control (like on Terminal Services) to fix user application issues.\n<\/p>\n<p>Both of these solutions drastically change the user system but they totally eliminate the effect of latency or bandwidth restrictions on cross-WAN or Internet application usage.\u00a0 I&#8217;ve used them in the past with great success.\n<\/p>\n<p><strong><u>Summary<\/u><\/strong>\n<\/p>\n<p>So that\u2019s a basic look at bandwidth VS latency and how they impact Internet and WAN based services.\u00a0 We saw how dedicated appliances, The Next Generation TCP Stack and how Windows 7 paired with Windows Server 2008 R2 can work to reduce bandwidth limitations as well as geography caused latency.\u00a0 The basic lesson is, look at more than just bandwidth.\u00a0 Without optimisation, latency will continue to negatively impact interactive services no matter how much expensive bandwidth you throw at a problem, e.g. you cannot make Sydney-Australia move any closer to Dublin-Ireland.\n<\/p>\n<p>EDIT #1: I added a section on Terminal Services and VDI.\n<\/p>\n<\/p>\n<\/li>\n<\/li>\n<\/li>\n<\/li>\n<\/li>\n<\/li>\n","protected":false},"excerpt":{"rendered":"<p>I\u2019ve done a little speaking about this subject over the last while and after listening to a recent radio conversation I thought I\u2019d post something too.\u00a0 The story was about a revolutionary online gaming system, the idea being that instead of buying a DVD with the game, you\u2019d play it online and it would stream &hellip; <a href=\"https:\/\/aidanfinn.com\/?p=9566\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Slow Network: Bandwidth VS Latency&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-9566","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"aioseo_notices":[],"jetpack_featured_media_url":"","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/9566","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9566"}],"version-history":[{"count":0,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/9566\/revisions"}],"wp:attachment":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9566"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9566"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9566"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}