A blog covering Azure, Hyper-V, Windows Server, desktop, systems management, deployment, and so on …
I do not know what the root cause of my location-specific outage last Friday was. I know that my Vodafone Ireland broadband at home was affected. I also know that Sky Ireland broadband was affected. But others internationally and the ISPs at work had no issues. It was all very strange … and the problem appears to have sorted itself out today (the following Wednesday).
Anywho, business (and sarcy posts) as normal!
Another slow 24 hours:
It was a quiet weekend. Note a useful scripts for health checking a Scale-Out File Server (SOFS) by Jose Barreto.
If you’re affected by this issue then you should have read this post. Microsoft posted a KB article for when virtual machines lose network connectivity when you use Broadcom NetXtreme 1-gigabit network adapters on Windows Server 2012 Hyper-V or Windows Server 2012 R2 Hyper-V.
Symptoms
When you have Hyper-V running on Microsoft Windows Server 2012 or Windows Server 2012 R2 together with Broadcom NetXtreme 1-gigabit network adapters (but not NetXtreme II network adapters), you may notice one or more of the following symptoms:
- Virtual machines may randomly lose network connectivity. The network adapter seems to be working in the virtual machine. However, you cannot ping or access network resources from the virtual machine. Restarting the virtual machine does not resolve the issue.
- You cannot ping or connect to a virtual machine from a remote computer.
These symptoms may occur on some or all virtual machines on the server that is running Hyper-V. Restarting the server immediately resolves network connectivity to all the virtual machines.
Cause
This is a known issue with Broadcom NetXtreme 1-gigabit network adapters that use the b57nd60a.sys driver when VMQ is enabled on the network adapter. (By default, VMQ is enabled.)
The latest versions of the driver are 16.2 and 16.4, depending on which OEM version that you are using or whether you are using the Broadcom driver version. Broadcom designates these driver versions as 57xx-based chipsets. They include 5714, 5715, 5717, 5718, 5719, 5720, 5721, 5722, 5723, and 5780.
These drivers are also sold under different model numbers by some server OEMs. HP sells these drivers under model numbers NC1xx, NC3xx, and NC7xx.
Workaround
Broadcom is aware of this issue and will release a driver update to resolve the issue. In the meantime, you can work around the issue by disabling VMQ on each affected Broadcom network adapter by using the Set-NetAdapterVmq Windows PowerShell command. For example, if you have a dual-port network adapter, and if the ports are named NIC 1 and NIC 2 in Windows, you would disable VMQ on each adapter by using the following commands:
Set-NetAdapterVmq -Name “NIC 1” -Enabled $False
Set-NetAdapterVmq -Name “NIC 2” -Enabled $FalseYou can confirm that VMQ is disabled on the correct network adapters by using the Get-NetAdapterVmq Windows PowerShell command.
Note By default, VMQ is disabled on the Hyper-V virtual switch for virtual machines that are using 1-gigabit network adapters. VMQ is enabled on a Hyper-V virtual switch only when the system is using 10-gigabit or faster network adapters. This means that by disabling VMQ on the Broadcom network adapter, you are not losing network performance or any other benefits because this is the default. However, you have to work around the driver issue.
Get-NetAdapterVmqQueue shows the virtual machine queues (VMQs) that are allocated on network adapters. You will not see any virtual machine queues that are allocated to 1-gigabit network adapters by default.
Sigh. I hope Broadcom are quicker about releasing a fix than Emulex (customers are waiting 10 or 11 months now?).
My site is hosted on Azure in the Dublin (Europe North) region. On Friday morning, I was checking something when I saw my site was not loading correctly – it was either offline or VERY slow. So I check the Azure status and saw it was offline. I restarted the application pool and the problem remained. I rebooted. MySQL took an age to load, but the site was still not loading … from home.
I have endpoint monitoring configured. Notice that Amsterdam was showing an issue and Chicago was not. Strange, eh? I’ve worked in hosting and I know how localised these problems can be. So it was time to start digging.
I asked online and people in Denmark were OK. Folks in Belfast and Netherlands had connection problems. Later, Denmark went offline and Amsterdam came back!
From Home (Vodafone Ireland – very slow/no access) I ran a tracert:
From the lab at work (Magnet ISP – access OK) I had different results:
From a VM with an ISP (Blacknight – access OK) I had different results again:
It was very odd. Nothing was red on the Azure status site. I’m guessing there was a localized issue within Azure that affected just a subset of us, or there was an external routing issue that affected some ISPs.
It’s still like this as I post … in other words, the site is fine for some and offline for others.
EDIT (30/7/2014):
I came home today to find that my site was once again available via my ISP.
Well done, Simon! You win this award because you:
Heck, 2 of the links were written by Microsoft, one by me, one on Hyper-V.nu and one by Thomas Maurer. We community contributors spend a lot of time writing this stuff. Please don’t expect us to read it to you too.
You can lead a horse to water but you cannot make him drink.
Congratulations to my MVP colleagues, Alessandro Cardoso and Benedict Berger on the recent publication of their respective books.
Alessandro wrote this second edition book to focus on SCVMM 2012 R2, available from Amazon:
Overview
- Create, deploy, and manage datacenters and private and hybrid clouds with hybrid hypervisors using VMM 2012 R2
- Integrate and manage fabric (compute, storages, gateways, and networking), services and resources, and deploy clusters from bare metal servers
- Explore VMM 2012 R2 features such as Windows 2012 R2 and SQL 2012 support, converged networks, network virtualization, live migration, Linux VMs, and resource throttling and availability
What you will learn from this book
- Plan and design a VMM architecture for real-world deployment
- Configure network virtualization, gateway integration, storage integration, resource throttling, and availability options
- Integrate SC Operations Manager (SCOM) with VMM to monitor your infrastructure
- Integrate SC APP Controller (SCAC) with VMM to manage private and public clouds (Azure)
- Deploy clusters with VMM Bare Metal
- Create and deploy virtual machines from templates
- Deploy a highly available VMM Management server
- Manage Hyper-V, VMware, and Citrix from VMM
- Upgrade from previous VMM versions
Benedict wrote this book, available from Amazon:
This is a step-by-step guide to implement best practice configurations from real-world scenarios, enhanced with practical examples, screenshots, and step-by-step explanations.This book is intended for those who already have some basic experience with Hyper-V and now want to gain additional capabilities and knowledge of Hyper-V.If you have used Hyper-V in a lab environment before and now want to close the knowledge gap to transfer your Hyper-V environment to production, this is the book for you!
Congratulations to both authors!
Before anyone asks – no, I am not planning an update to the WS2012 Hyper-V book. It’s too much work for too little return in too small a window (Windows Server vNext Preview will be announced in October, 12 months after the RTM of WS2012 R2).
Very little for you today:
Microsoft’s data centres are pretty “green”. And when I say green, I mean that they build & install only what they absolutely need, and they focus very heavily on power. A common measurement stick is Power Usage Effectiveness (PUE), which Wikipedia defines as:
… how much energy is used by the computing equipment (in contrast to cooling and other overhead).
The lower the number, the better. Microsoft does not share their PUE publicly, but according to the Green (Low Carbon) Data Center Blog the:
… PUE figures for its newest data centers which range from 1.13 to 1.2.
That’s an incredibly efficient achievement. I know quite a bit about how Microsoft do this, but I’m under NDA, after NDA, after NDA 🙂 All I can say is jump at the chance if you ever have an opportunity to tour on of the Microsoft Global Foundation Services modern data centres.
So what drives Microsoft? Sure, getting the likes of Greenpeace on your side is always good,especially when trying to sell business to environmentally sensitive customers. But the biggest reason for electrical efficiency is to save money. Electricity is only becoming more and more expensive. Data centers are growing in size and number, and are competing for this limited resource with each other, and with us (customers, consumers, businesses, etc). So saving a hundredth from a PUE figure could be worth millions of dollars every year (if not more!).
According to Fool.com, Microsoft has gone one step further by acquiring 20 years of power supply from a wind farm in Illinois, USA. This produces 175MW of power, all for Microsoft! And before that, Microsoft agreed to purchase 100% of production from a wind farm in Texas.
In theory, this is a renewable energy source with a pretty fixed cost. That contrasts nicely with competing for electricity from producers that are using dwindling carbon-based fuels. The strategy allows Microsoft to budget long-term, and it doesn’t hurt that renewable power will get a nod of approval from those wearing vegan trousers. It makes sense that Microsoft will continue this trend worldwide, thus making property costs and climate the only variations in the cost of operating Azure in different regions.
It was 8 months ago when I purchased my Lenovo Yoga Tablet 8, an 8” Android tablet. I raved about the form factor, price ($206.99 on Amazon.com, £150.99 on Amazon UK, €153.06 on Amazon Germany), and all that jazz.
So how has the tablet worked out?
I mentioned in my previous post that I was going to test the battery life in my upcoming travels. I did:
And that was before a firmware update that increase published battery life from 16 hours to 18 hours. I suspect that this device pulls power from dark matter in the universe. It is incredible, with only Kindle readers beating it.
The screen is not the best for viewing photos … but let’s be clear. The machine is CHEAP and works great for video.
I probably use this device more than any machine other than my PC at work. I travel with it, using it to keep myself entertained in hotels, airports, planes, etc. I keep it at my bedside locker, so I can check up on things when I hit the snooze button in the mornings. It has replaced my Kindle reader as my way of consuming books – the extra large battery doubles as a comfortable handle.
I’ve used a Micro-SD to expand the paltry 16 GB of inbuilt storage. Using a SD converter, I can quicky copy content from a PC/laptop onto the machine. Combined with the hotspot on my phone, I have easy Internet access. Throw in ProXPN and I am accessing Netflix USA while in Europe, and UK/Irish services while abroad. My Bose headphones give me perfect sound in a noisy environment.
The lightweight CPU has not been an issue for me. I don’t play many games – but Robocop, Plants VS Zombies 2, and the Angry Birds carting thing play fine.
I have a lot of good things to say about this device. I wish it was a Windows machine – I do have a Toshiba Encore tablet but the Yoga wins on battery life (against almost everything) and apps (quantity & quality VS Windows).
I strongly recommend this tablet to anyone needing an affordable mobile device, and who would like to complete their journey with some battery life left …. which is actually a big deal with airport security now.