Monitoring & Alerting for Windows Defender in Azure VMs

In this post, I will explain how one can monitor Windows Defender and create incidents for it with Azure VMs.

Background

Windows Defender is built into Windows Server 2016 and Windows Server 2019. It’s free and pretty decent. But it surprises me how many of my customers (all) choose Defender over third-parties for their Azure VMs … with no coaching/encouragement from me or my colleagues. There is an integration with the control plane using the antimalwareagent extension. But the level of management is poor-none. There is a Log Analytics solution, but solutions are deprecated and, last time I checked, it required the workspace to be in per-node pricing mode. So I needed something different to operationalise Windows Defender with Azure VMs.

Data

At work, we always deploy the Log Analytics extension with all VMs – along with the antimalware extension and a bunch of others. We also enable data collection in Azure Security Center. We use a single Log Analytics workspace to enable the correlation of data and easy reporting/management.

I recently found out that a table in Log Analytics called ProtectionStatus contains a “heartbeat” record for Windows Defender. Approximately every hour, a record is stored in this table for every VM running Windows Defender. In there, you’ll find some columns such as:

  • DeviceName: The computer name
  • ThreatStatusRank: A code indicating the health of the device according to defender:
    • 150: Health
    • 470: Unknown (no extension/Defender)
    • 350: Quarantined malware
    • 550: Active malware
  • ThreatStatus: A description for the above code
  • ThreatStatusDetails: A longer description
  • And more …

So you can see that you can search this table for malware infection records. First thing, though, is to filter out the machines/records reporting that there is no Defender (Linux machines, for example):

let all_windows_vms = Heartbeat

| where TimeGenerated > now(-7d)

| where OSType == ‘Windows’ | summarize makeset(Resource); ProtectionStatus

| where Resource in (all_windows_vms)

| sort by TimeGenerated desc

The above will find all active Windows VMs that have been reporting to Log Analytics via the extension heartbeat. Then we’ll store that data in a set, and search that set. Now we can extend that search, for example finding all machines with a non-healthy state (150):

let all_windows_vms = Heartbeat

| where TimeGenerated > now(-7d)

| where OSType == ‘Windows’

| summarize makeset(Resource); ProtectionStatus

| where Resource in (all_windows_vms)

| where ThreatStatusRank <> 150

| sort by TimeGenerated desc

Testing

All the tech content here will be useless without data. So you’ll need some data! Search for the Eicar test string/file and start “infecting” machines – be sure to let people know if there are people monitoring the environment first.

Security Center

Security Center will record incidents for you:

You will get email alerts if you have configured notifications in the subscription’s Security Center settings. Make sure the threshold is set to LOW.

If you want an alternative form of alert then you can use a Log Analytics alert (Scheduled Query Alert resource type) based on the below basic query:

SecurityAlert

| where TimeGenerated > now(-5m)

| where VendorName == ‘Microsoft Antimalware’

The above query will search for Windows Defender alerts stored in Log Analytics (by Security Center) in the last 5 minutes. If the threshold is freater than 0 then you can trigger an Azure Monitor Action Group to tell whomever or start whatever task you want.

Workbooks

Armed with the ability to query the ProtectionStatus table, you can create your own visualisations for easy reporting on Windows Defender across many machines.

 

The pie chart is made using this query:

let all_windows_vms = Heartbeat

| where TimeGenerated > now(-7d)

| where OSType == ‘Windows’

| summarize makeset(Resource); ProtectionStatus

| where TimeGenerated > now(-7d)

| where Resource in (all_windows_vms)

| where ThreatStatusRank <> ‘150’

| summarize count(Threat) by Threat

With some reading and practice, you can make a really fancy workbook.

Azure Sentinel

I have enabled the Entity Behavior preview.

Azure Sentinel is supposed to be the central place to monitor all security events, hunt for issues, and where to start investigations – that latter thanks to the new Entity Behavior feature. Azure Sentinel is powered by Log Analytics – if you have data in there then you can query that data, correlate it, and do some clever things.

We have a query that can search for malware incidents reported by Windows Defender. What we will do is create a new Analytic Rule that will run every 5 minutes using 5 minutes of data. If the results exceed 0 (threshold greater than 0) then we will create an incident.

let all_windows_vms = Heartbeat

| where TimeGenerated > now(-7d)

| where OSType == ‘Windows’

| summarize makeset(Resource); ProtectionStatus

| where TimeGenerated > now(-5m)

| where Resource in (all_windows_vms)

| where ThreatStatus <> ‘No threats detected’ or ThreatStatusRank <> ‘150’ or Threat <> ”

| sort by Resource asc | extend HostCustomEntity = Computer

The last line is used to identity an entity. Optionally, we can associate a logic app for an automated response. Once that first malware detection is found:

You can do the usual operational stuff with these incidents. Note that this data is recorded and your effectiveness as a security organisation is visible in the Security Efficiency Workbook in Azure Sentinel – even the watchers are watched! If you open an incident you can click investigate which opens a new Investigation screen that leverages the Entity Behavior data. In my case, the computer is the entity.

The break-out dialogs allow me to query Log Analytics to learn more about the machine and its state at the time and the state of Windows Defender. For example, I can see who was logged into the machine at that time and what processes were running. Pretty nice, eh?

The Office – Construction Complete

The construction of Cloud Mechanix global HQ finished yesterday afternoon. The final piece to go in place was the step up to the door. You can really see the slope in the site in the below photo.


If you step in you can see the all-wood finish, with the 1st fitting electrics, ready for the final touches.

And you can see the view from one of the locations where one of the desks will be located.

We had some paving stones from the site clearance, so they are being repurposed to cross the lawn from the house to the front door. A swing set was placed directly in-front of the office – the replacement is going to the far end of the lawn. The grass underneath and the empty anchor spots are bald so I’ll be re-sowing grass there in the coming days.

So what’s next? Paint and second-fitting electrics are next in the project plan. I will also be looking at how we can extend the house security systems to the office – a combination of a supplier-based monitored alarm system and the Ring cameras. That will allow us to insure the contents of the new office, and then … it’ll be time to move in.

The New Office – Day 3

It’s Monday and day 3 of the construction of Cloud Mechanix Global HQ. On Friday, the carpenters finished the steel roof and installed the wall studs for the cavity wall with insulation, as you can see in the photo.

One of the options we selected in the installation was to have a “plinth” and step installed. The price for this is always estimated on the site – the site effects the levelling and height of the foundation, and the height determines the amount of wood and labour required. The quote I was given was €350. The team is planning to finish the interior and additional wood work today. Next up will be the painters and electrical second fitting. We also added the options for paint and painting in the purchase. I can paint, but I suck at edges and I’m slow; I’d rather let a professional do it plus that means it’s time I can spend doing other things.

Thoughts are now turning to the interior, which will be painted white. Floorboards are being installed and also being painted white. But I am considering installing additional flooring to protect the structural wood. If I do, it’ll be something dark, or even grey, to contrast somewhat with the white walls and ceiling. I already have a white L-shaped Ikea “Bekant” desk. I think I will stick with the Ikea desks – I do like them. The plan is that the L-shaped desks will be placed behind the windows at the front, giving us a nice view up the garden while working. We’ll see where the budget is in a week or so, but I’m partial towards the version of the Bekant that includes powered legs that can raise/lower the desk at the push of a button.

I have an office chair already with a white frame and black cushions. But it was a bad purchase – the piston lasted only a few months and it makes a cracking noise when I adjust my position – that was why I replaced the previous seat! I can see myself buying some Secret Lab chairs so comments on that are welcome – from people who have owned one for more than 1 year.

I have a white 2×4 Ikea “Kallax” unit in the small office in the house at the moment. On the plus side, it’s great to display certain things. On the negative side, the close options are too small for the stuff you want to hide – I have 2 large plastic storage boxes hidden in the built-in wardrobe in the office at the moment, filled with things like spare peripherals, cables, battery packs, and so on. I think I want something that is closed storage at the bottom and open shelf storage at the top. I will keep looking.

My desk is quite full at the moment: 3 x USB drives (that I might replace with a NAS), a laptop, an Intel NUC, a KVM switch, a Surface dock, an Epson eco-tank printer, 2 x 27″ monitors, a mic, and so on. Some of the stuff is stuff that you need around but don’t need beside you all the time. Because we have a nearly 6 x 4m space, we have room to add more desk space. So I am thinking of putting in 1, or even 2, additional straight desks at a later time. Things like the printer and networking gear can be relocated to there.

Finally, there are times where you want to work, but want to relax. Maybe you need to read something in a more relaxed position, or just take a break from the screen. I’m thinking that we’ll put in a black leather couch in middle of the office – I’m not sure where yet because I might also move the 43″ TV from the house office into there – nice for watching conferences, etc. The second-hand market sometimes is great for that kind of thing – my mother-in-law is quite skilled at buying/selling on those kinds of websites so we might have to recruit her assistance. Depending on how that goes and how much space there is, we might add a small coffee table too.

The New Office – Construction Has Begun

It is day 2 of the construction of our new office. The carpenters arrived exactly at 9 am yesterday morning, bang-on time, to begin the assembly and construction of our new garden office – the global headquarters of Cloud Mechanix.

My only concern at the start was whether the site would be level enough for the build. The team warned us early on that the slope would make one side look higher up when it was level but we knew that and were OK with it. They quickly assembled the wooden frame that would eventually site on concrete blocks.

Once the 6x4m frame was built I was called out to pick the final position – allowing just over half a metre from the concrete garden wall so I can paint the outside to maintain the integrity of the wooden construction. Not long after the wooden frame was in position. Around lunchtime, the planks for the outside wall were being hammered into place. I’d say it took the 3 of them 45 minutes to get all the planks in place. Then the ceiling was put on.

The electricians were called in and asked to confirm how we wanted to do things after they surveyed the site. The one thing I was worried about was how the armoured electrical cable would run from the office to the fuse box at the front of the house – we need to power at least 2 computers + equipment and a 1.5 KW heater (please save comments on €13,000 hearing/aircon systems – the office is costing that much!). The ethernet line and electrical cable are entering on the back-right and will run along the garden wall under the capping (which you cannot see in the photos here). Then they will run underground to the front of the house. The ethernet line will enter the house at the same location as the phone/Internet and be just a meter from the broadband router. The electrical cable will cross the front of the house hidden in white trunking (maybe 2 cm wide) attached under the white soffit of the house – nicely camouflaged. The electrical cable will enter above the front door (still camouflaged) and then be under 1 metre from the fusebox – trunking from the front door, along an interior white doorframe – still camouflaged.

The electricians did the first fitting: 5 x dual power sockets, light switch, and 6 x dropdown lights. When leaving, they let me know that they would be back once the interior was finished by the carpenters to do the second fittings – sockets, lights, and the aforementioned cable installations.

The carpenters carried on, adding insulation and sheeting to the roof before the end of the day.

As you can see, there are two big windows, right where the front desks will be – one for me and one for my wife. Cloud Mechanix will have a great view overlooking about 40m of the garden!

Day two (today) began with drizzle. The team (down to two) were bang-on time again. They told me that they expect to require 3 days for the complete assembly and finishing of the wooden structure. Today, the plan was to get the doors and windows done, the studs for the wall insulation & interior installed, and the roof started. As you can see, the doors and windows were in place at lunchtime.

Exterior trims have been added, the slats for the roof (steel roof) are in place, and I can hear steel being cut as I type – they are installing the steel sheet roof that has a black tile effect. Once that is done, the slow bit (as they referred to it) will begin: the interior finishes, the lower trim and the step.

The Office Installation Is About To Start

It’s a while (about 6 weeks) since I wrote that I am getting a new office built in the garden. Today seems like a good time to post an update.

There were 3 birch trees on the site where the office is being installed. I had the trees cut down (we have already replaced them with 4 apple trees and 2 more will be added next year) and that left the stumps. If I had been clever, I would have hired a mini-digger and dug around them. Maybe I would have also hired a tree grinder to chop up the revealed stumps. But no – instead we decided to dig them out by hand. Birch trees have interesting roots. The root ball is about 60-80 CM deep, and roots, sometimes 3x wider (or more) than the trunk, come out and up towards the surface. Those roots break out into a fine mesh that spreads out around the site, holding the soil firmly together, almost like concrete. I had a pick-axe, an axe, a shovel, a trowel, and my bare hands. The pick-axe was literally bouncing off the ground. The roots were like a phone book, and a sharp axe wasn’t cutting; it was breaking a layer or two and bounding too. I’ll keep this short – the final stump took over 8 hours to dig out by myself, and ended with me using my hands to scoop out dirt from under the root ball and roll the stump until the tap-root snapped.

Tip: hire the machinery, and don’t do it by hand.

The next thing to deal with was the base. Once the trees were removed, we realised that the site had a slant that we had never noticed before – about 27cm from left to right over 6m. I thought that would require a concrete base. So I put out some feelers to find a builder that would install the base. I got 4 names. Only 3 returned my contact attempts. Of the 3, only 2 bothered to show up – one of those was 2 days late! And of the 2, neither either bothered to give me a quote. I guess that builders are busy enough, even with all the COVID disruption that’s been going on. I called the cabin builders to get their opinion – and they said that we didn’t need a base. Their method of foundation installation would handle the minor slope. Perfect!

The electrician was the next thing. After my last post, I had lots of people telling me to go spend another 13,000-20,000 Euros on different kinds of heating systems – me arse! We didn’t even spend that on the house! I got an electrician quote – a guy who came recommended from the cabin builders. I’d tried reaching out to one other electrician but he was as useless at responding as the builders. The recommended guy is booked and will be doing the first (armored cable from the fusebox) and second fittings, as well as an Ethernet wire installation from our broadband router in the house to the office.

The last piece of the installation will be painting – that’s paid for with the installation and will be done afterward.

So when will all this happen? A lorry dropped off an “Ikea-like” load in our driveway a couple of hours ago. An installation team is due tomorrow at 9am to start the assembly. Hopefully we won’t have to wait too long for the painters and then we can get the alarm/security stuff installed, a new floor surface installed, and purchase some new office furniture.

Started Building A New Home Office

As I have posted before, I work from home. For a number of reasons, my wife and I have decided that I should move the office out of the house and into our garden. So that means that I need a new building in the garden. I spent months reading over the different options and providers. Eventually, I settled on one provider and option.

I’ve chosen to build a “log cabin”, actually a modern wooden building, in our back garden. I’ve chosen all the upgrades for thicker walls, wall/floor/roof insulation, upgraded roof, guttering & drainage, and so on, making it a warm place to work in the winter. If I had to explain it, I’d say it’s Nordic in design. The building is 6 x 4 meters. I went big because it gives me lots of space (better to have more than needed than retreat a smaller choice later) and gives my wife the option of from home too. We paid the deposit last week and are expecting delivery and installation in 5-7 weeks.

The first challenge was where to put the new office. Many of my American friends won’t think that a 6×4 meter building is big or that space shouldn’t be a problem. In Europe, that is big and space is very much a problem. However, I’m lucky, because our back garden was an accident of poor planning by the builder and ended up being ~65 meters long. We were lucky when we bought the house 4 years ago because the previous owner possibly undervalued the site – he didn’t sell through an estate agent.  We have lots of space, most of it hilly, but we have space. After some discussions, it was decided to put the office near the house at the bottom of the garden for 3 reasons:

  • Networking convenience
  • Power convenience
  • The view

The view question was interesting. From the rear of the house, the office will be mostly out of view, with the living parts of the house still looking out onto the back garden. The office will have windows and glass doors on the front, overlooking the garden. So depending on my seating angle, I will be looking out over the green view of the back garden. But there are a few issues.

The first was that there were three nearly-20-years-old birch trees in the exact corner that the office will be going into. I hated the thought of tearing down those trees. But the corner is not used – the trees create a haven for flies and my kids won’t go in there to play – that was my vision for the area. On the other hand, the left side of our garden is lined with a variety of mature trees, we’ve planted 4 more 18 months ago and they are doing well, and both sides are surrounded by trees outside our border wall. So we called up the local handyman who cut down the trees and removed the cuttings – he uses the wood after drying it out.

That was the easy bit! The tree stumps had to be removed next. You can’t just cut/grind a stump down and hope that’s that. Nature is tough. The roots would live on and a new tree could grow through the floor of the office. You have to cut all the roots from the trunk, get under it and lift it out. So out came shovels, an axe and a pickaxe. After 5 hours, 2 of the 3 stumps were removed on Saturday. That was back-breaking work. It turns out that the birch tree grows roots down deep. Those roots are in tick fibrous branches that quickly break out into a mesh of fibres that spread out 3cm to 30 cm deep, protecting the soil around the tree from tools such as shovels and pickaxes. You have to dig, tear and cut just to get through that first few CM of soil and then you face the roots that an axe will bend, not cut. And when you think you’ve cut the last root that is securing the trunk, you find that there are more. It’s Monday now and I’m facing 1 last stump to remove. It’s only in the last few hours that stiffness from Saturday has set in – so today will be fun!

Cutting down the trees revealed something that we had not noticed. The site where the office will reside – about 7m wide, is not level. There is a ~30cm slope going from left to right, and a lesser uneven slope from front to back. The office must be installed onto a level site. I evaluated the options – I hoped that I could dig out one side and use the soil to level out the other side. But that would mean digging under the boundary wall and weakening the foundations. There is no option other than to build a concrete base or pad. At my wife’s suggestion, I went onto the local community page on Facebook and asked for builder recommendations. I reached out to 4 builders – 2 are coming today to give me a quote and one is to call me about making an appointment. I’ll need the pad installed ASAP – concrete sets quickly but takes weeks to harden.

And finally … there’s the electrical installation – something that I know that I cannot do. The cabin manufacturers recommended an electrician. 10 double sockets, lights, a 1.5KW storage heater and connection to the fuse box will clock in at a sizeable sum of change, plus VAT (sales tax). We’ll try to get some alternative quotes for that next.

Why A Bastion Host Is Necessary For Remote VM Administration (Including Azure)

This post will explain why you should use a “Bastion Host” or a “Jump Box” to securely remote into Linux (SSH) or Windows (Remote Desktop) virtual machines. And this advice also includes machines that you run in a cloud, such as Microsoft Azure.

For the Fundamentalists on Social Media

Some people are going to make some comments like:

“This is why you should use remote Bash|PowerShell scripting”

Or maybe:

“You should be using Windows Admin Center”.

Windows Admin Center – great! Genuinely. But it does not do everything.

There are still many times when you need to directly log into a machine and do something; that’s real life, and not some blogger’s lab life.

Security Center JIT VM Access?

I was a fan of this feature. That was until they changed how the allow (RDP, SSH, etc) rules were added to an NSG. In my work, every subnet is micro-segmented. That means that the last user-defined NSG rule is Deny All from * to *. Since JIT VM Access was changed, it moves the last rule (if necessary) and puts in the allow-RDP or all-SSH (or whatever) rule after the DenyAll rule which is useless. Feedback on this has been ignored.

Why Are SSH and RDP Insecure?

I can’t comment too much on SSH because I’m allergic to penguins. But I can comment on RDP. Over the last few months, I can think of 3 security alerts that have been released about pre-authentication vulnerabilities that have been found in Remote Desktop. What does that mean?

Let’s say that you have a PC on your WAN that is infected by malware that leverages a known or zero-day pre-authentication remote desktop vulnerability. If that PC has the ability to communicate with a remote VM, such as an Azure Windows/Linux VM, via SSH or RDP then that remote machine is vulnerable to a pre-authentication attack. That means that if malware gets onto your network, and that malware scans the network for open TCP 22 or TCP 3389 ports, it will attempt to use the vulnerability to compromise the remote VM. It does not require the user of the PC to SSH or RDP into the remote VM, or to even have any guest OS access! You can put a firewall in front of the remote virtual machines, but it will do no good; it’s still allowing TCP 3389 or TCP 22 directly into the virtual machines and all it will offer is logging of the attack.

A Bastion Host

You might have heard the term “bastion” in the Azure world recently. However, the terms Bastion Host or Jump Box are far from new. They’re an old concept that allows you to isolate valuable machines and services behind a firewall but still have a way to remote into them.

The valuable remote virtual machines are placed behind a firewall. In Azure, that could be a firewall appliance, such as Azure Firewall, and/or Network Security Groups. Now to connect to the remote VMs, you must first remote into the Bastion Host. And from that machine, you will remote further into the network through the isolation of the firewall/NSGs.

But that’s still not perfect, is it? If we do simple SSH or RDP to the Bastion Host, then it is vulnerable to pre-authentication attacks. And that means once that machine is compromised, it can attack further into the remote network. What we need is some kind of transformation.

Remote Desktop Gateway

My preferred solution is to deploy a Remote Desktop Gateway (RDGW) as the bastion host – this does not require RDP licensing for administrative access to the remote virtual machines! The Bastion Host is deployed as one virtual machine or 2+ load-balanced virtual machines that allow in HTTPS connections via firewall/NSG rules. When an administrator/developer/operator needs to log into a remote VM, their Remote Desktop client is configured to connect to this gateway using HTTPS instead of RDP. Once the connection is authenticated by the RDGW, it reverse proxies the connection through to the desired virtual machine, further protected by firewall/NSG rules. Now the malware that is on the WAN cannot probe any machines in the remote network; there is no opening across the network to TCP 3389 or TCP 22. Instead, the only port open for remote connections is HTTPS which requires authentication. And internally, that transforms to connections from the RDGW to the remote VMs via TCP 3389.

Some sharp-eyed observers might notice that the recently announced CVE-2020-0609  is a pre-authentication attack on RDGW! Yes, unpatched RDGW deployments are vulnerable, but they are smaller in number and easier to manage patches for than a larger number of other machines. Best practice for any secure network is to limit all external ports. Transforming the protocol in some way, like an RDGW, further reduces the threat of that single opening to a single service that forwards the connection.

If you want to add bells and whistles, you can deploy Network Policy Server(s) to centrally manage RDGW policy and even add multi-factor authentication (MFA) via Azure AD.

This is great for Windows, but what about Linux? I’m told that Guacamole does a nice job there. However, Guacamole is not suitable for recent releases of Windows because of how it must have hardcoded admin credentials for Network Layer Authentication (NLA).

Azure Bastion

Azure Bastion made lots of noise in IT news sites, and on blogs and social media when it went into preview last year, and eventually it went GA at Ignite in November of last year. Azure Bastion is a platform-based RDGW. Today (January 2020), I find it way too limited to use in anything but the simplest of Azure deployments:

  • The remote desktop authentication/connection are both driven via the Azure Portal, which assumes that the person connecting into the guest OS even has rights to the Azure resources.
  • It does not support desktop Remote Desktop/SSH clients.
  • It does not offer MFA support for the guest OS login, only for the Azure Portal login (see above).
  • VNet peering is not supported, limiting Azure Bastion to pretty simple Virtual Network designs.

If Azure Bastion adds VNet peering, it will make it usable for many more customers. If it understands that guest OS/Azure resource rights OS/Azure Portal logins can be different, then it will be ready for mid-large enterprise.

 

The Worst IT Project I Was Ever A Part Off

This post will discuss a failed project that I was brought into and what I observed and learned from that project. It’s a real scenario that happened years ago, involving an on-premises deployment.

Too Many Chefs Spoil the Broth

Back in 2010, I joined a Dublin-based services company. The stated intention from the MD was that I was to lead a new Microsoft infrastructure consulting team. As it turned out, not a single manager or salesperson in the company believed that there was any work out there in Microsoft infrastructure technology – really! – and it never really got off the ground. But I was brought into one customer, and this post is the story of that engagement.

It was a sunny, cold day when I drove out to the customer’s campus. They are a large state-owned … hmm … transport company. I had enough experience in IT to know that I was going to be dealing with strong personalities and opinions that were not necessarily based on fact. My brief was that I would be attending a meeting with all the participants of a failing Windows Server 2008 R2 Hyper-V and System Center 2008 R2 project. I came in and met a customer representative – the technical lead of the project. He immediately told me that I was to sit in the corner, observe, and not talk to any of the participants from the other services providers. Note that the last word is plural, very plural.

I sat at the far corner of a long board room table and in came everyone. There was the customer IT managers and tech staff, the storage manufacturer (HP – now HPE) and their partner, the networking manufacturer (Cisco) and their partner, a Microsoft Premier Field Engineer, the consultants that implemented Hyper-V, the consultants that implemented the System Center management of Hyper-V, and probably more. Before I continue, I think that the Hyper-V cluster was something like 6 nodes and maybe 50-100 VMs.

Quickly it became evident that this was the first time that any of the participants in the meeting had talked to each other. I should re-phrase that: this was the first time any of the participants in deploying the next-generation IT infrastructure for running the business had been allowed to talk to each other.

  • A new W2008 R2 Hyper-V cluster was built. Although the customer was adamant that this was not true (it was), a 2-site cluster was built as a single-site cluster. There was a latent link between the two sites, with no control of VM placement and no third-site witness.
  • HP P4000 “Lefthand” module-based iSCSI storage was used without any consideration of persistent iSCSI reservations – a common problem in the W2008 R2 era, where the volumes in the SAN would “disappear” from the cluster because the scale-out of NICs/CSVs/SAN nodes went beyond the limits of W2008 R2 – a result of poor understanding of storage performance and Hyper-V architecture.
  • I remember awful problems with backup. DPM was deployed by a consulting firm but configured by a local staff member. He had a nightmare with VSS providers (HP were awful at this) and backup job sizing. It was not helped by the fact that backup was an afterthought in Hyper-V back then, not resolved really until WS2012 when it became software-defined. This combined with how the P4000 worked, the multi-site cluster that wasn’t, and Redirected IO caused all sorts of fun.
  • VMs would disappear – yup the security officer insisted that AV was installed on each host and it scanned every folder, including the CSVs. They even resisted change when presented with the MS documentation on scan exceptions that must be configured on Windows Server roles/features, including Hyper-V.

These were just a few of the technical issues; there were many more – inconsistent or missing patching, NIC teaming issues, and so on. I even created a 2-hour presentation based on this project that I (unofficially) called “How to screw up a Hyper-V project”.

My role was to “observe” but I wanted this thing fixed, so I contributed. I remember I spent a lot of time with the MS PFE on the customer site. He was gathering logs on behalf of support and we shared notes. Together we identified many issues/solutions. I remember one day, the customer lead shouted at me and ordered me back to my desk. I was not there “to talk to people but to observe”. The fact that I was one of two people on site that could solve the issues was lost on him.

The customer’s idea of running a project was to divide it up into little boxes and keep everyone from talking to each other. Part of this was how they funded the project – once it went over a certain monetary level it had to be publicly tendered. They had their preferred vendors and they went with them, even if they were not the best people. This created islands of knowledge/expertise and a lack of a vision. The customer thought they could manage this, and they were wrong. Instead, each supplier/vendor did their own thing based on assumptions of what others were doing and based on incorrect information shared by the customer’s technical team. And it all blew up in the customer’s face.

In the end, I heard that the customer blamed the software, the implementors, and everyone else involved in the project but themselves. They scrapped the lot and went with VMware, allegedly.

Lessons Learned

I think that there were three major lessons to be learned from this project. I know that these lessons apply equally today, no matter what sort of IT project you are doing, including on-premises, hybrid, or pure cloud.

The Business

IT enables or breaks the business. That’s something that most boards/owners do not understand. They think of IT as the nerds playing Doom in a basement, with their flashing lights and whirring toys. Obviously, that’s a wrong opinion.

When IT works, it can make the IT faster, more agile, and more competitive. New practices, be they operational or planning, can change IT, but I’ve even read how SCRUM/Agile concepts can even be brought to business planning.

Any significant IT project that will impact the business must start with the business. Someone at the C-Level must own it, be invested in it, and provide the rails or mission statement that directs it. That oversight will force those involved in the project to operate correctly and give them guidance on how to best serve the business.

Architecture

Taking some large-impact IT project and treating it as a point solution will not work. For example, building an entirely new IT infrastructure without considering the impact of or the dependencies on networking is stupid! You cannot just hand-off systems to different vendors and wish them bon voyage. There must be a unified vision. This technical vision starts with the previously mentioned business vision that guide-rails the technical design. All components that interconnect and have direct/indirect involvements must be designed as a whole.

Unified Delivery

The worst thing one can do is divvy up IT infrastructure to 5 or 6 vendors and say, you do that, and I will participate in a monthly meeting. That’s not IT! That’s bailing out on your responsibility! IT vendors can play a role, when chosen well. But they need a complete vision to do their job. And if they cannot get that from you, they must be allowed to help you build it. If your IT department’s role is to manage outsourcing contracts and nothing more, you have already failed the business and should just step aside.

A unified delivery must start with internal guidance, sharing the complete vision with all included parties, internal and external, as early as possible. Revealing significant change that you are working on with Vendor A 6 months into a project with Vendor B is a fail. Isolating each of the vendors is a fail. Not giving each vendor clear rules of engagement with orchestrated interaction is a fail. The delivery must be unified under the guidance of the architect who has a complete vision.

Bad IT Starts at The Top

In my years, I’ve done plenty of projects, reviewed many customer’s IT systems, and worked as a part of IT departments. Some of them were completely shocking. A common theme was the CIO/CTO: typically, an accountant or finance officer who was handed the role of supervising IT because … well … it’s just IT and they have a budget to manage. Someone who doesn’t understand IT, hires/keeps bad IT managers, and bad IT managers hire bad IT staff, make bad IT decisions, and run bad IT projects. As the saying goes, sh&t rolls downhill. When these bad projects are happening to you, and you run IT, then you must look at the mirror and stop pointing the finger elsewhere.

And before you say it, yes, there are crap consultants too 😊

 

 

Working From Home – The Half Year Update

It’s July now, and it’s just over half a year (just over 7 months to be more accurate) since I started a new job where I work from home. Here are some thoughts and lessons from my experience.

The Change

In my previous job, I was mostly in the office. I had a certain amount of flexibility where I could work from home to get things done in peace and quiet, away from the noise of an open-plan office, but mostly I commuted (by car, because there is no public transport option from here to the outskirts of Dublin where most private sector employees work) with my wife every morning and evening. We worked together – having met at the office – we were “podmates” for years before she moved desks and we started going out. I was happy in that job, and then along came Innofactor Norway who offered me a job. A big thing for me was not to travel – we have a young family and I don’t want to miss life. It was agreed and I changed jobs, now working as a Principal Consultant, working mostly with Azure networking and security with large clients and some internal stuff too.

Work Environment

This is not my first time having a work-from-home job. I worked with a small company before where they eliminated the office from expenses to keep costs down. It suited everyone from the MD down that we could work from home or on the road – one salesman used to camp out in a hotel in west Dublin, order coffee all day, and use their WiFi while having meetings close to potential customers’ offices. I worked from my home in the midlands. I was single then and worked long hours. I remember one evening when I’d ordered a pizza. The poor delivery guy was stuck talking to me, who hadn’t spoke to anyone in a week, and was trying to get away from this overly chatty weirdo. I had a sort-of-office; it was a mess. I ended up getting into the habit of working from the sitting room – which was not good. It was uncomfortable and there were entertainment distractions.

And important part of this job was going to be the work environment. I have an office in the house, but had rarely used it. A lot of my writing work for Petri or even writing courses for Cloud Mechanix was actually done on the road, sometimes literally while in the car and waiting for my daughter outside ballet or gymnastics classes. The office needed work so I cleared it out completely, painted and redecorated, and then set up all the compute from scratch. A nice fast PC with dual monitors was isntalled. I added a smart TV from the new year’s sales to the wall for YouTube/conference streams, as well as an extra screen (the MS adapter is much better than built-in screen casting). For work I bought a new Surface Laptop 2 (black, 16 GB RAM, i7) with the Surface Dock. And the folks in my old job gave me a present of a StarTech 2 port KVM switch, to allow me to share the keyboard and monitors between my NUC PC and Surface Laptop. The Laptop is my primary machine with Microsoft Whiteboard and the pen being hugely important presenting tools. The desk in the above picture is gone – it was purchased 3 years ago as a temporary desk when we moved into our house. I replaced it with a white IKEA L-shaped desk because the old one was squeaky and annoying me during Teams calls. Next to be replaced will be the chair.

Teams & Browser

The two tools I use the most are Microsoft Teams and the browser.

Teams is the primary communications tool at work. Very little internal comms use email. Email tends to be for external comms or some sort of broadcast type of message. I work with another Irish MVP, Damian Flynn. We’ve worked on a couple of big projects together, which has been awesome, and it’s not unusual for us to have a window open to each other for most of the day, even if we’re working on different things. It’s like we’re “podmates” but with half the width of Ireland between us. I’ve really noticed the absence this week – Damian is on vacation and most of Norway is on vacation too. I’m starting my second week with barely any conversation during the workday.

Working in cloud, obviously the browser is important. I started off in Chrome. Then I discovered identity containers in Firefox and I switched. However, I had to use Edge too – our time keeping is (how we bill each hour) is done in CRM and it won’t work in a non-MS browser. I was having problems with Teams randomly dropping calls. I even was considering switching to Slack/Skype (consumer) for my work with Damian. Then I realised that I was running out of RAM and that was killing Teams calls. The culprit? Firefox was eating many GBs of RAM. I had started to play with the Chromium Edge preview (aka ChrEdge) and was impressed by the smoothness of it (particularly scrolling for reading). When I realised how it had implemented identities (which is not perfect) I made the switch. Now ChrEdge is the only browser running all of the time and Teams is not losing calls.

Of course, I do a lot of JSON, so VS Code is nearly always open all of the time too. With that, Git client, and Azure DevOps, I have a way of doing things at scale, repeatedly, and collaboratively.

Work Habits

I am quite strict with myself, as if I am in an office, even though “my office” is half way up the west coast of Norway. I’ve heard presenters on home working talk about getting on a bike or in a car to drive around the block and “come to the office” with a work mentality. I’ll be honest, I might not even put on pants in the morning (kidding!) but I do clock in at 8:00 every morning. By the way, Norwegians start work before most of Europe is even awake. I’m the “late riser” in the company. Sure it’s 9am Norwegian time when I start, but they’ve already been at work for hours when I start. I have a routine in the morning, getting the kids ready, having breakfast, and bringing a coffee into my office. I have a morning start routine when there isn’t an immediate call, and then I’m straight into it. My lunch routine involves a bike ride or walk, weather permitting, and a quick lunch, and then I work the rest of my hours. I try to work longer hours at the start of the week so I can finish early on Friday – it’s amazing how a 2 day weekend feels like 3 days when you can finish just a little bit earlier.

More Than Just Work

Being at home means I’m also here for my kids. I can pick up our youngest earlier from day care. Our eldest can be brought to training or football games – and for checkups after breaking her arm – without taking time off from work. All are nice perks from working from home and having flexible hours/employer. This week, my wife is away at Microsoft Inspire. Without the time waste of a commute, I can use that time to do work around the house and keep on top of things. Soon enough, our youngest will start pre-school and I will be able to bring her there and back again. And when school starts, she’ll be walked to the end of our road, where the school is, and back again. Even when things are normal, there are times when I’ll have dinner started before my wife gets home – my culinary knowledge and skills are limited so I won’t subject my poor family to me cooking every day!

I’ve mentioned that I am using my lunch break for exercise. I’m overweight – like many in the business – a byproduct of sitting 8-10 hours a day. I started cycling 2-3 times a week spring of last year and put the bike away for the winter. It came out again this spring, and I was on it 2 times per week. II needed more time on it, so I’ve started going out every day. Sometimes I’ll do a long/fast walk to break it up, giving me a chance to catch up on podcasts and audio books.

More Productive & Less Stress

With less time wasted in traffic jams and more time focused on doing productive work, I am sure that I am more productive at home. Study after study have documented how bad open plan offices are. In the quiet of my office, I can focus and get stuff done. In fact, this time lead to me doing some business changing work with Damian. I’m less stressed than ever too.

If you can find a way to work, have reliable broadband, and have an employer that has realized how a happier employee is a more productive one, then I couldn’t recommend this style of working enough.

 

I’m Speaking At IP Expo in Manchester This Week

I will be in Manchester, UK, this week. I will be presenting an updated version of my “Solving the Azure Storage Maze” talk during the Altaro-sponsored slot at 12:45 on Wednesday in the Infrastructure Modernisation Theatre of IP Expo – registration is free.

If you have ever struggled with understanding all the storage options (not including databases – because I have 30 minutes, not 1 day) in Azure then I will help you navigate through all of the options. Note that I have updated this talk with Premium Blob and Premium Files content.