An IT Person On A Weight Loss Journey

In this post, I’m going to share a little about my weight loss journey – I’m an IT pro and the sedentary lifestyle isn’t exactly great for staying trim!

Let’s go back a few years. Back in 2016-2017 I was weighing in at around 16 stone 10 pounds (106 KG or 234 lbs); that’s a great weight for a 6’3″ NFL linebacker, not a 5’7″ IT consultant.

In the above picture with Paula Januszkiewicz, I might have been talking about new security features in Hyper-V, but I was probably wondering what pizza I might order at the hotel when I wrapped up.

Living the Good Life

When I grew up, there wasn’t a lot of money for unhealthy food. Food was homemade, sometimes even homegrown. Treats like chocolate might happen once every few weeks and during the holidays. There were no regular trips to “the chipper” for a burger, pizza didn’t exist … it was eat healthily or don’t eat.

Back when I was in school/college, I cycled to and from school and I was stick-thin at 11 stone 7 pounds – I know that’s heavy for someone my height but I was actually stick thin.

Then along came graduation, my first job in IT, and money. Every mid-morning, there was a trip to the company canteen (for a fried breakfast), followed by lunch, followed by a convenient dinner. I swear I could eat takeaway 7 days a week. I worked in international consulting, living in hotels for 6 months at a go, so I ate a lot either in the hotel or in a bar/restaurant. I remember sitting down and thinking “what’s up with my waist?” when it first rolled over my belt.  And so it went for 20+ years.

I got a real shock 4-5 years ago when I had to do a health check for some government stuff, and I was classified as … obese. The shock! That only happens to Americans that dine at KFC! It was time for a change. By now, I had a 38″ waist.

Exercise

I have done some time in the gym over the years. During one of my 6-month hotel stints (in a small wealthy town called Knutsford near Manchester, UK) I spent an hour in the gym on most days. And then I ate a 2-3 course dinner with a bottle of wine. When I exercise, I get HUNGRY. Back in the late 2000’s I spent 5-7 days a week in the gym for over half a year. I’d do up to 90 minutes of training, followed by a 14″ pizza, chicken, and fries. I’m not kidding about the hungry thing!

I live near a canal and a lot of work was being done to open up a “greenway”  that’s a path dedicated to walkers, runners, and cyclists. Not long after the obese determination, I purchased a bike and started cycling, getting out several times a week to do a good stretch. That first year I did quite a bit on the bike. I felt fitter, but not smaller. I came home and ate the same way as always. I drank lots of beer and wine.

2 years ago I was in Florida with my family for a 3-week vacation. I didn’t have enough shorts with me so I went to a nearby outlet and tried on my usual 38″ waistline. Huh! They were too … big! I ended up buying 36″ shorts and jeans that vacation and they fit just nice.

Last year, I decided that I needed to do something more. My diet needed work. I wasn’t getting time to go out on the bike anymore – a wife and kids that need me around and the pending arrival of a new baby (twins as it turned out) would kill off the hours away on the bike at a time. I joined a group that runs a simplified calorie counting scheme. The goal is that you make adjustments to reduce your calorie intake but still get to enjoy the nice things – but within limits. I seriously started at that program in August of last year. At that time I weighed 15 stone 12 pounds (100 KG or 222 lbs), not bad for an NFL running back.

Food Optimisation

The program I’m using doesn’t use the term “diet”. Diet means starving yourself. In this program, I can still eat, but the focus is on including more salad/veg (1/3 of my plate), swapping out wasteful calories, and replacing oils and butter. It has resulted in myself and my wife experimenting so much more with our cooking – even I cook, which is probably quite a shock to the takeaway industry. The results were fast. Soon all my 38″ waistline clothes went into recycling. In June of this year, I was a 34″ waist and weighing 12 stone 12 pounds (81 kg or 180 lbs), which is pretty OK for a small NFL wide receiver. This is where things went a little wrong when I was last in the gym, 10 years ago. My weight never would go lower – the 14″ pizzas (etc) didn’t help.

I miss beer 🙂 I used to love getting craft beer and trying it out. I switched over to lower-calorie whiskey & dark rum. But then my wife spotted that 0 alcohol beer has very few calories. It might be missing the buzz, but the taste is still there – perfect while cooking on the BBQ.

Running != Weight Loss

While on maternity leave this summer, my wife started walking. I tried to get out a bit on the bike but was really restricted to once or twice at the weekend. I started to join my wife on her walks on Saturdays and Sundays. We pushed the pace and really started clocking up the KMs, varying between 7-14km per walk. I could feel the difference in my fitness level as we kept walking faster.

I struggled a bit again. The summer months brought lots of good weather and I was making the most amazing homemade burgers on the BBQ, along with steak, and chicken … you get the idea. My weight was bouncing down and up.

And then my wife spotted that the gym at a local hotel was running a family deal. We could sign up the entire family, the kids could use the pool, and me and my wife could use the gym. I went the next day and I used the bike and I ran. I ran 5 KMs with a few stretches of walking to get a drink. I was shocked. Now when I say that I ran, I was crawling along at 5 kmph – not exactly Raheem Mostert burning up the injusry-causing turf on the New York Jets last year.

I ran for a week. I was a little sore. I weighed in the following week and I was up 2 lbs! What the f***? There were a few things that I realised/learned with some googling:

  • When you start/change your exercise routines, you “damage” muscle (it’s part of the building muscle process) and that causes your body to retain fluid which temporarily increases your weight.
  • Running does not cause weight loss alone. You need to still have a calorie deficit. Following up a run by pigging out does not help.
  • Muscle (which I can feel) adds weight but can cause calorie “afterburn” so you burn off more calories even when resting.

Where I Am Now

I set myself a goal of hitting 12 stone (168 lbs or 76 kg). I’m also aiming to get my waistline below 30″ – an important milestone in long-term health. I now weigh around 12 stone 7 lbs (175 lbs or 79 Kgs) and my waist is 32″. My weight is trending down over a 2-3 week period and I’m running 5 KMs, now at 9.5 kmph with sprints up to 13 kmph. If I have time, I’ll combine weights and bike with running. If I hit the 12 stone mark and I’m still carrying a few too many pounds, then I’ll aim for 11 stone 7 pounds and see how things go.

Me about a month ago.

I’m Not Missing Out – Much

I used to live on pizza. I miss it 🙂 We had a week away back in June and I gave myself a week off from the food optimisation. I ate fried food and pizza – a whole pizza – to myself. I drank nice beer every night. And my weight did go up by 4 lbs, but I lost it all within 10 days.

As I said, we’re cooking a lot. There are lots of websites and cookbooks that specialise in making nice food but with better ingredients. Last week I make double cheeseburgers using low-fat cheese and beef and they were DE-F’N-LICIOUS. We’ve been eating curries, air-fried everything, and slow cooking stuff like mad. Our fridge is full of sauces and our press is stuffed with spices. We’re at the point where restaurant food has to be pretty amazing to impress us now.

We like this series of books by an Irish couple that became well known on Instagram – they actually live near us and we bumped into them eating dinner around the corner from our home.

Amazon UK


Amazon US

 We’re also fans of the Pinch of Nom series:

Amazon UK


Amazon US

Feeling Like You Could Lose Weight?

I’m not an expert. The best advice I can give you is:

  • Start now. Make the decision. Go Bo Jackson and just do it.
  • Find the right food optimisation technique for you. I like the one I’m using because it’s flexible and isn’t about “punishing” you. My wife’s uncle did something different and shed weight in no time.
  • Exercise. This will help burn the calories. When combined with the calorie deficit of your food optimisation, you’ll see a difference.
  • Be patient. Weight loss is different for everyone. For some, there’s an instant buzz when the pounds go off. For others, they are shocked when lots of exercise leads to weight gain! Be consistent, do the right things, and be patient. It’s not about what happens today, it’s what happens over a 1, 2, 6, 12month period.

But most of all – enjoy it. I gained weight out of laziness and because I enjoyed certain foods. Now, I’m eating healthier than I ever did but I am still enjoying food – the chicken fillet burgers I had a couple of weeks ago might appear on the menu tomorrow (better than any I ever had out) and I’m looking forward to the beef curry that we’re making tonight!

Testing Azure Firewall IDPS

In this post, I will show you how to test IDPS in Azure Firewall Premium, including test exploits and how to search the logs for alerts.

Azure Firewall Setup

You are going to need a few things:

  • Ideally a hub and spoke deployment of some kind, with a virtual machine in two different spokes. My lab is Azure Virtual WAN, using a VNet as the “compromised on-premises” and a second VNet as the target.
  • Azure Firewall Premium SKU with logging enabled to a Log Analytics Workspace.
  • Azure Firewall Policy Premium SKU, with IDPS enabled for Alert & Deny.

Make sure that you have firewall rules and NSG rules open to allow your “attacks” – the point of IDPS is to stop traffic on legitimate protocols/ports.

Compromised On-Premises Machine

One can use Kali Linux from the Azure Marketplace but I prefer to work in Windows. So I deployed a Windows Server VM and downloaded/deployed Metasploit Opensource, which is installed into C:\metasploit-framework.

The console that you’ll use to run the commands is C:\metasploit-framework\bin\msfconsole.bat.

If you want to trying something simpler, then all you will need is the normal Windows Command prompt.

The Exploit Test

If you are using Metasploit, in the console, run the following to search for “coldfusion” tests:

search coldfusion

Select a test:

use auxiliary/scanner/http/coldfusion_locale_traversal

Set the RHOST (remote host to target) option:

set RHOST <IP address to target>

Verify that all required options are set:

show options

Execute the test:

run

Otherwise, you can run the following CURL command in Windows Command Prompt for a simpler test to do a web request to your target IP using the well-known Blacksun user agent:

curl -A “BlackSun” <IP address to target>

Check Your Logs

It can take a little time for data to appear in your logs. Give it a few minutes and then run this query in Log Analytics:

AzureDiagnostics | where ResourceType == “AZUREFIREWALLS” | where OperationName == “AzureFirewallIDSLog” | parse msg_s with Protocol ” request from” SourceIP “:” SourcePort ” to ” TargetIP “:” TargetPort “. Action:” Action”. Signature: ” Signature “. IDS:” Reason | project TimeGenerated, Protocol, SourceIP, SourcePort, TargetIP, TargetPort, Action, Signature, Reason | sort by TimeGenerated

That should highlight anything that IDPS alerted on & denied – and can also be useful for creating incidents in Azure Sentinel.

Understanding the Azure Virtual Desktop Resources

In this post, I will document the resources used in Azure Virtual Desktop, what they do, and how they interconnect.

This is a work-in-progress, so any updates I discover along the way will be added. You should also check out a similar post on Azure Image Builder.

Host Pool – Microsoft.DesktopVirtualization/hostpools

The host pool documents the configuration of the hosts that will provide the desktops/applications. Note that a Host Pool resource ID is required to create an Application Group.

Note, the VMs themselves are deployed using a linked template when you use the Azure Portal. My deployment used the “managed disks” template. This template deploys the VMs, runs some DSC, joins the machines to your domain. There is also a task to update the host pool.

The result of running Microsoft.DesktopVirtualization/hostpools does not create the VMs – it just manages any VMs added to the Host Pool.

The mandatory properties appear to be:

  • hostPoolType: BYODesktop, Personal, or Pooled.
  • loadBalancerType: BreadthFirst, DepthFirst, or Persistent.
  • preferredAppGroupType: Persistent, None, or RailApplications.

The full set of properties can be found in the REST API documentation.

Application Group – Microsoft.DesktopVirtualization/applicationgroups

The Application Group documents the applications, user associations (the Desktop Virtualization User role is assigned to users/groups), and is associated with a Host Pool; therefore you must deploy a Host Pool resource before you deploy the planned Application Group.

The mandatory values appear to be:

  • hostPoolArmPath: The resource ID of the associated Host Pool
  • applicationGroupType: Desktop or RemoteApp

We know that Windows 365 (AKA “Cloud PC”) is built on Azure Virtual Desktop. Proof of that is in ARM, with a true/false property called cloudPcResource.

The REST API documentation has complete documentation of the properties.

Workspace – Microsoft.DesktopVirtualization/workspaces

The Azure Virtual Desktop Workspace is the glue that holds everything together. The Workspace can be associated with no, 1, or many Application Groups via a non-mandatory array value called applicationGroupReferences. You can build a Workspace before your Application Groups and update this value later. Or you can build the (1) Host Pool(s), (2) Application Group(s), followed by the Workspace.

The mandatory values appear to be:

  • applicationGroupReferences: An array value with 0+ items, each being the resource ID of an Application Group.

    Virtual Machines

The Host Pool will require virtual machines; these are created as a separate deployment. There’s nothing special here; they are virtual machines created from the Marketplace or from your own generalised image (captured or Shared Image Gallery). Two actions must be done to the VMs:

  • Domain Join: Either (legacy) ADDS (including Azure AD DS or Windows Server ADDS) or an Azure AD Join (a recent feature add).
  • Virtual Desktop agent: DSC will be used to deploy the agent. This will make an outbound connection to the Host Pool and register the VM.

AAD, AADDS, or ADDS? I prefer ADDS. This is because:

  • Most of the controls that you need are in Group Policy and AAD doesn’t do Group Policy.
  • AADDS relies on AAD which is a single-region service. If that region has AAD issues (and this happens pretty frequently) then your Azure Virtual Desktop farm is dead.
  • Third-party applications typically expect ADDS and will not support AADDS/AAD, even if it “works”.

Service Definitions in Azure Firewall

In this post, I will discuss how you can use Rules Collection Groups in Azure Firewall to aggregate your Rules Collections and Rules to be aligned with a service definition or workload definition.

Workload and Service Definitions

In an organised environment, every workload (or service) has a definition. That definition describes all of the components that make up and facilitate the workload. That includes your firewall rules.

So it would make sense if you had a way of grouping firewall rules together if those rules are used to make a workload possible. For example, if I was running an Azure Virtual Desktop pool, I might treat that pool as a workload, document it as a workload, and want all of the rules that make that pool possible to be grouped together and managed as a unit.

Challenges

The challenges I have faced with Azure Firewall and aligning rules along the model of a workload definition have been:

Realisation & Conceptualisation

I’ve always used some kind of workload-based approach but my approach wasn’t perfect. I decided to try to align rules with NSGs, placing inbound rules with the workload that was the destination. But then some workloads, such as Windows Admin Center or ADDS, require more reach, and then you get messy. And what if you have a dozen or more workloads that are atomic units but are also extremely integrated? Where do the rules go?

I’ve come to realise that rules should go with the service that they empower, regardless of destination. What made me think like that? Documentation of workloads for a forced-ad-hoc migration project that I’ve been working on for the last year. We didn’t get the chance to assess and plan in detail, so everything was an on-the-fly discovery. Our method of “place the rule with the target workload” has created very complicated ARM templates for Azure Firewall; defining a workload from a firewall perspective is very hard with that approach.

If we said that “all network rules that empower a workload go with the workload’s network Rules Collection” then things would get a bit better – a bit.

Rules Groups Collections Limitations

It is a year since Firewall Policy became generally available. Just like last year, I had some hours to experiment on Azure Firewall. I tried out the new tier, Rules Collection Groups. A reminder:

  1. Rules > Rules Collections (typed, based on DNAT, Network, or Application)
  2. Rules Collections > Rules Collection Groups

But at the time, the new tech was immature. Mixing Rules Collection types between Rules Groups was a disaster. The advice I got from the product group was “don’t do it, it’s not ready, stick with the default groups for now”.

So I did just that. That means that the DNAT rules collection,  the Network Rules Collection, and the Application Rules Collection for any workload were split into 3 deployments, the default rules group collections for:

  • DNAT
  • Network
  • Application

Each of those deployments could have dozens or hundreds of rules collections for each workload. And when you combine that with my previous approach to rules placement:

  1. It’s a mess
  2. Deploying a rule change required re-deploying all Rules Collections of a type for all workloads in using that type of Rules Collection.

A Better Approach

I have had some time to play and things are better.

Rules Alignment With Workload

I’ve discussed this already – place rules that empower a workload with the workload, not the destination workload.

If Workload X requires TCP 445 access to Workload Y, then I will create a Network Rule to Workload Y on TCP 445 in a Network Rules Collection for Workload X. The result is that all rules that make Workload X function will be in rules collections for Workload X. That makes documentation easier and makes the next step work.

Rules Collection Groups For Workload

This is the big change in Azure from this time last year. I can now create lots of Rules Collection Groups, each with a priority (for processing order). I will create 1 Rules Collection Group per workload. Workload X will get 1 Rules Collection Group.

All rules that make Workload X go into Rules Collection Groups for Workload X. I might have, depending on rules requirements, up to 6 Rules Collection Groups:

  • DNAT-Allow-WorkloadX
  • Network-Allow-WorkloadX
  • Application-Allow-WorkloadX
  • DNAT-Deny-WorkloadX
  • Network-Deny-WorkloadX
  • Application-Deny-WorkloadX

The Rules Collection Group is its own Deployment from an ARM perspective. If I’m managing the firewall as code (I do) then I can have 1 template (or parameters file) that defines the Rules Collection Group (and contained Rules Collections and Rules) for the entire workload and just the workload. Each workload will have its own template or parameter file. A change to a workload definition will affect 1 file only, and require 1 deployment only.

If you want to see an ARM template for deploying one of the workloads in my screenshot, then head on over to my GitHub.

Wrap Up

This approach should leave the firewall much better organised, easier to manage in smaller chunks if using infrastructure-as-code, be easier to document, and more suitable for organisations that like to create/maintain service definitions.

Defending Against Supply Chain Attacks

In this post, I will discuss the concepts of supply chain attacks and some thoughts around defending against them.

What Is A Supply Channel Attack?

The recent ransomware attack through Kaseya made the news but the concept of a supply chain attack isn’t new at all. Without doing any research I can think of two other examples:

  • SolarWinds: In December 2020, attackers used compromised code in SolarWinds monitoring solutions to compromise customers of SolarWinds.
  • RSA: In 2011, the Chinese PLA (or hackers sponsored by them) compromised RSA and used that access to attack customers of RSA.

What is a supply chain attack? It’s pretty hard to break into a network, especially one that has hardened itself. Users can be educated – ok, some will never be educated! Networks can be hardened and micro-segmented. Identity protections such as MFA and threat detection can be put in place.

But there remains a weakness – or several of them. There’s always a way into a network – the third party! Even the most secure network deployments require some kind of monitoring system – something where a piece of software is deployed onto “every VM”.  Or there’s some software vendor that’s deep into your network that has openings all over the place. Those are your threats. If an attacker compromises the software from one of those vendors then they will get into your network during your next update and they will use the existing firewall holes & permissions that are required by the software to probe, spread, and attack.

Protection

You still need to have your first lines of defense, ideally using tools that are designed for protection against advanced persistent threats – not your regular AV package, dumby:

  1. Identity
  2. Email
  3. Firewall
  4. Backup with isolated offline storage protected by MFA

That’s a start, but, but a supply chain attack bypasses all that by using existing channels to enter your network as if it is from a trusted source – because the attack is embedded in the code from a trusted source.

Micro-Segmentation

The first step should be micro-segmentation (AKA multi-segementation). No two nodes on your network should be able to communicate unless:

  1. They have to
  2. They are restricted to the required directions, protocols, and ports.
  3. That traffic passes through a firewall – and ideally several firewalls.

In Microsoft Azure, that means using:

  • A central firewall, in the form of a network firewall and/or web application firewall (Azure or NVA). This firewall controls connections between the outside world and your workloads, between your workloads, and importantly from your workloads to the outside world (prevents malware from talking to its human controller).
  • Network Security Groups at the subnet level that protect the subnet and even isolate nodes inside the subnet (use a custom Deny All rule because the default Deny All rule is useless when you understand the logic of how it works).
  • Resource firewalls – that’s the guest OS firewall and Azure resource firewalls.

If you have a Windows ADDS domain, use Group Policy to force the use of Windows Firewall – lazy admins and those very same vendors that will be the channel of attack will be the first to attempt to disable the firewall on machines that they are working on.

For Azure resources, consider the use of Azure Policy to force/audit the use of the firewalls in your resources and a default route to 0.0.0.0/0 via your central firewall.

An infrastructure-as-code approach to the central firewall (Azure Firewall) and NSGs brings documentation, change control, and rollback to network security.

Security Monitoring

This is where most organisations fail, and even where IT security officers really don’t get it.

Syslog is not security monitoring. Your AV is not security monitoring. You need something bigger, that is automated, and can filter through the noise – I regularly use the term “be your Neo to read the Matrix”. That’s because even in a small network, there is a lot of noise. Something needs to filter through that noise and identity the threats.

For example, there’s a lot of TCP 445 connection attempts coming from one IP address. Or there are lots of failed attempts to sign in as a user from one IP address. Or there are lots of failed connections logged by NSG rules. Or even better – all of the above. These are the sorts of things that malware that is attempting to spread will do. This is the sort of work that Azure Sentinel is perfect for – Sentinel connects to many data sources, pulls that data to a central place where complex queries can be run to look for threats that a human won’t be able to do. Threats can create incidents, incidents can trigger automated flows to eliminate the noise, and the remaining incidents can create alerts that humans will act upon.

But some malware is clever and not so noisy. The malware that hit the HSE (the Irish national health service) uses a lot of manual control to quietly spread over a very long time. Restricting outbound access to the Internet to just the required connections for business needs will cripple this control mechanism. But there’s still an automated element to this malware.

Other things to implement in Azure will include:

  • IDPS: An intrusion detection & prevention in the firewall, for example Azure Firewall Premium. When known malware/attack flows pass through the firewall, the firewall can log an alert or alert/deny the flows.
  • Security Center: Enabling Security Center “Azure Defender” (the tier previously known as the Azure Security Center Standard) provides you with oodles of new features, including some endpoint protections that are very confusingly packaged and licensed by Microsoft.

Managed Services Providers

MSPs are a part of the supply chain for their customers. MSP staff typically have credentials that allow them into many customer networks/services. That makes the identities of those staff very valuable.

A managed service provider should be a leader in identity security process, tooling, and governance. In the Microsoft world, that means using Azure AD Premium with MFA enabled for all staff. In the Azure world, Lighthouse should be used to gain access to customers’ cloud implementations. And that access should be zero-trust, powered by Privileged Identity Management (PIM).

Oh Cr@p!

These attackers are not script kiddies. They are professional organisations with big budgets, very skilled programmers and operators, and a lot of time and will. They know that with some persistent effort targeting a vendor, they can enter a lot of networks with ease. Hitting a systems management company, or more scarily, a security vendor, reaps BIG rewards because we invest in these products to secure our entire networks. The other big worry is those vendors that are deeply embedded with certain verticals such as finance or government. Imagine a vendor that is in every branch of a national government – one successful attack could bring down that entire government after  a wave of upgrades! Or hitting a well known payment vendor could open up every bank in the EU.

Understanding the Azure Image Builder Resources

In this post, I will explain the roles of and links/connections between the various resources used by Azure Image Builder.

Background

I enjoy the month of July. My customers, all in the Nordics, are off for the entire month and I am working. This year has been a crazy busy one so far, so there has been almost no time in the lab – noticeable I’m sure by my lack of writing. But this month, if all goes to plan, I will have plenty of time in the lab. As I type, a pipeline is deploying a very large lab for me. While that runs, I’ve been doing some hands on lab work.

Recently I helped develop and use an image building process, based on Packer, to regularly create images for a Citrix farm hosted in Microsoft Azure. It’s a pretty sweet solution that is driven from Azure DevOps and results in a very automated deployment that requires little work to update app versions or add/remove apps. At the time, I quickly evaluated Azure Image Builder (also based on Packer but still in Preview back then) but I thought it was too complicated and would still require the same pieces as our Packer solution. But I did decide to come back to Azure Image Builder when there was time (today) and have another look.

The first mission – figure out the resource complexity (compared to Packer by itself).

The Resources

I believe that one of Microsoft’s failings when documenting these services is their inability to explain the functions of the resources and how they work together. Working primarily in ARM templates, I get to see that stuff (a little). I’ve always felt that understanding the underlying system helps with understanding the solution – it was that way with Hyper-V and that continues with Azure.

Managed Identity – Microsoft.ManagedIdentity/userAssignedIdentities

A managed identity will be used by an Image Template to authorise Packer to use the imaging process that you are building. A custom role is associated with this Managed Identity, granting Packer rights to the resource group that the Shared Image Gallery, Image Definition, and Image Template are stored in.

Shared Image Gallery – Microsoft.Compute/galleries/images

The Shared Image Gallery is the management resource for images. The only notable attribute in the deployment is the name of the resource, which sadly, is similar to things like Storage Accounts in lacking standardisation with the rest of Microsoft Azure resource naming.

Image Definition- Microsoft.Compute/galleries/images

The Image Definition documents your image as you would like to present it to your “customers”.

The Image Definition is associated with the Shared Image Gallery by naming. If your Shared Image Gallery was named “myGallery” then an image definition called “myImage” would actually be named as “myGallery/myImage”.

The properties document things including:

  • VM generation
  • OS type
  • Generalised or not
  • How you will brand the images build from the Image Definition

Image Template – Microsoft.VirtualMachineImages/imageTemplates

This is where you will end up spending most of your time while operating the imaging process over time.

The Image Template describes to Packer (hidden by Azure) how it will build your image:

  • Identity points to the resource ID of the Managed Identity, permitting Packer to sign in as that identity/receiving its rights when using this Image Template to build an Image Version.
  • Properties:
    • Source: The base image from the Azure Marketplace to start the build with.
    • Customize: The tasks that can be run, including PowerShell scripts that can be downloaded, to customise the image, including installing software, configuring the OS, patching and rebooting.
    • Distribute: Here you associate the Image Template with an Image Definition, referencing the resource ID of the desired Image Definition. Everytime you run this Image Template, a new Image Version of the Image Definition will be created.

Image Version – Microsoft.Compute/galleries/images/versions

An Image Version, a resource with a messy resource name that will break your naming standards, is created when you build from an Image Template. The name of the Image Version is based on the name of the Image Definition plus an incremental number. If my Image Definition is named “myGallery/myImage” then the Image Version will be named “myGallery/myImage/<unique number>”.

The properties of this resource include a publishing profile, documenting to what regions an image is replicated and how it is stored.

What Is Not Covered

Packer will create a resource group and virtual machine (and associated resources) to build the new image. The way that the virtual machine is networked (public IP address by default) can normally be manipulated by the Image Template when using Packer.

Summary

There is a lot more here than with a simple run of Packer. But, Azure Image Builder provides a lot more functionality for making images available to “customers” across an enterprise-scale deployment; that’s really where all the complexity comes from and I guess “releasing” is something that Microsoft knows a lot about.

 

Building Azure VM Images Using Packer & Azure Files

In this post, I will explain how I am using a freeware package called Packer to create SYSPREPed/generalised templates for Citrix Cloud / Windows Virtual Desktop (WVD) – including installing application/software packages from Azure Files.

My Requirement

Sometimes you need an image that you can quickly deploy. Maybe it’s for a scaled-out or highly-available VM-based application. Maybe it’s for a Citrix/Windows Virtual Desktop worker pool. You just need a golden image that you will update frequently (such as for Windows Updates) and be able to bring online quickly.

One approach is to deploy a Marketplace image into your application and then use some deployment engine to install the software. That might work in some scenarios, but not well (or at all) in WVD or Citrix Cloud scenarios.

A different, and more classic approach, is to build a golden image that has everything installed and then the  VM is generalised to create an image file. That image file can be used to create new VMs – this is what Citrix Cloud requires.

Options

You can use classic OS deployment tools as a part of the solution. Some of us will find familiarty in these tools but:

  • Don’t waste your time with staff under the age of 40
  • These tools aren’t meant for the cloud – you’ll have to daisy chain lots of moving parts, and that means complex failure/troubleshooting.

Maybe you read about Azure Image Builder? Surely, using a native image building service is the way to go? Unfortunately: no. AIB is a preview, driven by scripting, and it fails by being too complex. But if you dig into AIB, you’ll learn that it is based on a tool called Packer.

Packer

Packer, a free tool from Hashicorp, the people behind Terraform, is a simple command line tool that will allow you to build VM images on a number of platforms, including Azure ARM. The process is simple:

  • You build a JSON file that describes the image building process.
  • You run packer.exe to ingest that JSON file and it builds the image for you on your platform of choice.

And that’s it! You can keep it simple and run Packer on a PC or a VM. You can go crazy and build a DevOps routine around Packer.

Terminology

There are some terms you will want to know:

  • Builders: These are the types of builds that Packer can do – the platforms that it can build on. Azure ARM is the one I have used, but there’s a more complex/faster Builder for Azure called chroot that uses an existing build VM to build directly into a managed disk. Azure ARM builds a temporary VM, configures the OS, generalises it, and converts it into an image.
  • Provisioners: These are steps in the build process that are used to customise your operating system. In the Windows world, you are going to use the PowerShell provisioner a lot. You’ll find other built in provisioners for Ansible, Puppet, Chef, Windows Restart and more.
  • Custom/Community Provisioners: You can build additional provisioners. There is even a community of provisioners.

Accursed Examples

If you search for Windows Packer JSON Files, you are going to find the same file over and over. I did. Blog posts, powerpoints, training materials, community events – they all used the same example: Deploy Windows, install IIS, capture an image. Seriously, who is ever going to want an image that is that simple?

My Requirement

I wanted to build a golden image, a template, for a Citrix worker pool, running in Azure and managed by Citrix Cloud. The build needs to be monthly, receiving the latest Windows Updates and application upgrades. The solution should be independent of the network and not require any file servers.

Azure Files

The last point is easy to deal with: I put the application packages into Azure Files. Each installation is wrapped in a simple PowerShell script. That means I can enable a PowerShell provisioner to run multiple scripts:

      “type”: “powershell”,
      “scripts”: [
        “install-adobeReader.ps1
        “install-office365ProPlus.ps1”
      ]
This example requires that the two scripts listed in the array are in the same folder as packer.exe. Each script is run in turn, sequentially.

Unverified Executables

But what if one of those scripts, like Office, wants to run a .exe file from Azure Files? You will find that the script will stall while a dialog “appears” (to no one) on the build VM stating that “we can’t verify this file” and waits for a human (that will never see the dialog) to confirm execution. One might think “run unlock-file” but that will not work with Azure Files. We need to update HKEY_CURRENT_USER (which will be erased by SYSPREP) to truse EXE files from the FQDN of the Azure Fils share. There are two steps to this, which we solve by running another PowerShell provisioner:
    {
      “type”: “powershell”,
      “scripts”: [
        “permit-drive.ps1”
      ]
    },
That script will run two pieces of code. The first will add the FQDN of the Azure Files share to Trusted Sites in Internet Options:

set-location “HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\Domains”
new-item “windows.net”
set-location “Windows.net”
new-item “myshare.file.core”
set-location “myshare.file.core”
new-itemproperty . -Name https -Value 2 -Type DWORD

The second piece of code will trust .EXE files:

set-location “HKCU:\Software\Microsoft\Windows\CurrentVersion\Policies”
new-item “Associations”
set-location “Associations”
new-itemproperty . -Name LowRiskFileTypes -Value ‘.exe’ -Type STRING

SYSPREP Stalls

This one wrecked my head. I used an inline PowerShell provisioner to add Windows roles & features:

      “type”: “powershell”,
      “inline”: [
        “while ((Get-Service RdAgent).Status -ne ‘Running’) { Start-Sleep -s 5 }”,
        “while ((Get-Service WindowsAzureGuestAgent).Status -ne ‘Running’) { Start-Sleep -s 5 }”,
        “Install-WindowsFeature -Name Server-Media-Foundation,Remote-Assistance,RDS-RD-Server -IncludeAllSubFeature”
      ]
But then the Sysprep task at the end of the JSON file stalled. Later I realised that I should have done a reboot after my roles/features add. And for safe measure, I also put one in before the Sysprep:
    {
      “type”: “windows-restart”
    },
You might want to run Windows Update – I’d recommend it at the start (to patch the OS) and at the end (to patch Microsoft software and catch any missing OS updates). Grab a copy of the community Windows-Update provisioner and place it in the same folder as Packer.exe. Then add this provisioner to your JSON – I like how you can prevent certain updates with the query:
    {
      “type”: “windows-update”,
      “search_criteria”: “IsInstalled=0”,
      “filters”: [
        “exclude:$_.Title -like ‘*Preview*'”,
        “include:$true”
      ]
    },

Summary

Why I like Packer is that it is simple. You don’t need to be a genius to make it work. What I don’t like is the lack of original documentation. That means there can be a curve to getting started. But once you are working, the tool is simple and extensible.

Enable FQDN-Based Network Rules In Azure Firewall

In this post, I will discuss how the DNS features, DNS Servers and DNS Proxy, can be used to enable FQDN-based rules in the Azure Firewall.

Can’t Azure Firewall Already Do FQDN-based Rules?

Yes – and no. One of the rules types is Application Rules, which control outbound access to services which can be based on a URI (a DNS name) for HTTP/S and SQL (including SQL Server, Azure SQL, etc) services. But this feature is not much use if:

  • You have some service in one of your VNets that needs to make an outbound connection on TCP 25 to something like smtp.office365.com.
  • You need to allow an inbound connection to an FQDN, such as a platform resource that is network-connected using Private Endpoint.

FQDN-Based Network Rules

Network rules allow you to control flows in/out from source to destinations on a particular protocol and source/destination port. Originally this was, and out of the box this is, done using IPv4 addresses/CIDRs. But what if I need to have some network-connected service reach out to smtp.office365.com to send an email? What IP address is that? Well, it’s lots of addresses:

nslookup smtp.office365.com

Non-authoritative answer:
Name: DUB-efz.ms-acdc.office.com
Addresses: 2603:1026:c02:301e::2
2603:1026:c02:2860::2
2603:1026:6:29::2
2603:1026:c02:4010::2
52.97.183.130
40.101.72.162
40.101.72.242
40.101.72.130
Aliases: smtp.office365.com
outlook.office365.com
outlook.ha.office365.com
outlook.ms-acdc.office.com

And that list of addresses probably changes all of the time – so do you want to manage that in your firewall(s) rules and in the code/configuration of your application? It would be better to use the abtsraction provided by the FQDN.

Network Rules allow you to do this now, but you must first enable DNS in the firewall.

Azure Firewall DNS

With this feature enabled, the Azure Firewall can support FQDNs in the Network Rules, opening up the possibility of using any of the supported protocol/port combinations, expanding your name-based rules beyond just HTTP/S and SQL.

By default, the Azure Firewall will use Azure DNS. That’s “OK” for traffic that will only ever be outbound and simple. But life is not normally that simple unless you host a relatively simple point solution behind your firewall. In reality:

  • You might want to let on-premises/remote locations connect to Private Endpoint-enabled PaaS services via site-to-site networking.
  • You might hit an interesting issue, which I will explain in a moment.

Before I move on, for the Private Endpoint scenario:

  1. Configure DNS servers (VMs) on you VNet configuration
  2. Configure conditional forwarders for each Private Endpoint DNS Zone to forward to Azure Private DNS via 168.63.129.16, the virtual public IP address that is used to facilitate a communication channel to Azure platform resources.
  3. Set the Azure Firewall DNS Server settings to point at these DNS servers
  4. Route traffic from your site-to-site gateway(s) to the firewall.

Split DNS Results

If two different machines attempt to resolve smtp.office365.com they will end up with different IP addresses – as you can see in the below diagram.

The result is that the client attempts to connect to smtp.office365.com on IP address A, and the firewall is permitting access on IP address B, so the connection attempt is denied.

DNS Proxy

To overcome this split DNS result (the Firewall and client getting two different resolved IP addresses for the FQDN) we can use DNS Proxy.

The implementation is actually pretty simple:

  1. Your firewall is set up to use your preferred DNS server(s).
  2. You enable DNS Proxy (a simple on/off setting).
  3. You configure your VNet/resources to use the Azure Firewall as their DNS server.

What happens now? Every time your resource attempts to resolve a DNS FQDN, it will send the request to the Azure Firewall. The Azure Firewall proxies/relays the request to your DNS server(s) and waits for the result which is cached. Now when your resource attempts to reach smtp.office365.com, it will use the IP address that the firewall has already cached. Simples!

And yes, this works perfectly well with Active Directory Domain Controllers running as DNS servers in a VNet or spoke VNet as long as your NSG rules allow the firewall to be a DNS client.

Recording – Introducing Azure Virtual WAN

Here is a video recording that I recorded last week called Introducing Azure Virtual WAN.

I was scheduled to do a live presentation for the (UK) Northern Azure User Group (NAUG). All was looking good … until my wife went into labour 5 weeks early! We welcomed healthy twin girls and my wife is doing well – all are home now. But at the time, I was clocking up lots of miles to visit my wife and new daughters in the evening. The scheduled online user group meeting was going to clash with one of my visits.

I reached out to the organiser, Matthew Bradley (a really good and smart guy – and someone who should be an MVP IMO), and explained the situation. I offered to record my presentation for the user group. So that’s what I did – I deliberately did a 1-take recording and didn’t do the usual editing to clean up mistakes, coughs, actually’s and hmms. I felt that the raw recording would be more like what I would be like if I was live.

The feedback was positive and I was asked if I would share the video. So here you go:

An Introduction to Azure ExpressRoute Architecture

This post will give you an overview of Azure ExpressRoute architecture. This is not a “how to” post; instead, the purpose of this post is to document the options for architecting connectivity with Microsoft Azure in one concise (as much as possible) document.

Introduction to ExpressRoute

Azure ExpressRoute is a form of private Layer-2 or Layer-3 network connectivity between a customer’s on-premises network(s) and a virtual network hosted in Microsoft Azure. ExpressRoute is one of the 2 Azure-offered solutions (also, VPN) for achieving a private network connection.

There are 2 vendor types that can connect you to Azure using ExpressRoute:

  • Exchange provider: Has an ExpressRoute circuit in their data centre. Either you run your “on-premises” in their data centre or you connect to their data centre.
  • Network service provider: You get a connection to an ISP and they relay you to a Microsoft edge data centre or POP.

The locations of ExpressRoute and Azure are often confused. A connection using ExpressRoute, at a very high level and from your perspective, has three pieces:

  • Circuit: A connection to a Microsoft edge data centre or pop. This can be one of many global locations that are often nothing to do with Azure regions; they are connected to the same Microsoft WAN as Azure (and Microsoft 365) and are a means to relay you to Azure (or Microsoft 365) using Azure ExpressRoute.
  • Connection: Connecting an Azure Virtual Network (ExpressRoute Gateway) in an Azure region to a circuit that terminates at the edge data centre or POP.
  • Peering: Configuring the routing across the circuit and connection.

For example, a customer in Eindhoven, Netherlands might have an ExpressRoute circuit that connects to “Amsterdam”; This POP or edge data centre is probably in Amsterdam, Netherlands, or the suburbs. The customer might use that circuit to connect to Azure West Europe, colloquially called “Amsterdam”, but is actually in Middenmeer, approximately 60 KM north of Amsterdam.

ExpressRoute Versus VPN

The choice between ExpressRoute and site-to-site VPN isn’t always as clear-cut as one might think: “big organisations go with ExpressRoute and small/mid go with VPN”. Very often, organisations are choosing to access Azure services over the Internet using HTTPS, with small amounts of legacy traffic traversing a private connection. In this case, VPN is perfect. But when you want an SLA or low latency, ExpressRoute is your choice.

Site-to-Site VPN ExpressRoute
Microsoft SLA Microsoft: Azure

Internet: No one

Microsoft: Azure

Service Provider: Circuit

Max bandwidth Aggregate of 10 Gbps 100 Gbps
Routing BGP (even if you don’t use/enable it) BGP
Latency Internet Low
Multi-Site See SD-WAN (Azure Virtual WAN) Global Reach

Also see Azure Virtual WAN

Connections Azure Virtual Networks Azure Virtual Networks

Other Azure Services

Microsoft 365

Dynamics 365

Other clouds, depending on service provider

Payment Outbound data transfer and your regular Internet connection Payment to service provider for the circuit.

Payment for either a metered (outbound data + circuit) or unlimited data (circuit) to Microsoft.

Terminology

  • Customer premises equipment (CPE) or Customer edge routers (CEs): 2, ideally, edge devices that will be connected in a highly available way to 2 lines connecting your network(s) to the service provider.
  • Provider edge routers (PEs), CE facing: Routers or switches operated by the service provider that the customer is connected to.
  • Provider edge routers (PEs), MSEE facing: Routers or switches operated by the service provider that connect to Microsoft’s MSEEs.
  • Microsoft Enterprise Edge (MSEE) routers: Routers in the Microsoft POP or edge data centre that the service provider has connected to.

The MSEE is what:

  • Your ExpressRoute virtual network gateway connects to.
  • Propagates BGP routes to your virtual network.
  • Can connect two virtual networks together (with BGP propagation) if they both connect to the same circuit (MSEE).
  • Can relay you to other Azure services or other Microsoft cloud services.

It is very strongly recommended that the customer deploys two highly available pieces of hardware for the CEs. The ExpressRoute virtual network gateway is also HA, but if the Azure region supports it, spread the two nodes across different availability zones for a higher level of availability.

FYI, these POPs or Edge Data Centers also host other Azure services for edge services.

Peering

Quite often, the primary use case for Azure ExpressRoute is to connect to Azure virtual networks, and resources connected to those virtual networks such as:

  • Virtual machines
  • VNet integrated SKUs such as App Service Environment, API Management, and SQL Managed Instance
  • Platform services supporting Private Endpoint

That connectivity is provided by Azure Private Peering. However, you can also connect to other Microsoft services using Microsoft Peering:

To use Microsoft Peering you will need to configure NAT to convert connections from private IP addresses to public IP addresses before they enter the Microsoft network.

ExpressRoute And VPN

There are two scenarios where ExpressRoute and site-to-site VPN can coexist to connect the same on-premises network and virtual network.

The first is for failover. If you deploy a /27 or larger GatewaySubnet then that subnet can contain an ExpressRoute Virtual Network Gateway and a VPN Virtual Network Gateway. You can then configure ExpressRoute and VPN to connect the same on-premises and Azure networks. The scenario here is that the VPN tunnel will be an automated failover connection for the ExpressRoute circuit – failover will happen automatically with less than 10 packets being lost. Two things immediately come to mind:

  • Use a different ISP for Internet/VPN connection than used for ExpressRoute
  • Both connections must propagate the same on-premises networks.

An interesting new twist was announced recently for Virtual Network Gateway and Azure Virtual WAN. By default, there is no encryption on your ExpressRoute circuit (more on this later). You will be able to initiate a site-to-site VPN connection across the ExpressRoute circuit to a VPN Virtual Network Gateway that is in the same GatewaySubnet as the ExpressRoute Virtual Network Gateway, encrypting your traffic.

ExpressRoute Tiers

There are three tiers of ExpressRoute circuit that you can deploy in Microsoft Azure. I have not found a good comparison table, so the below will not be complete:

Standard Premium
Price Normal More Expensive
Azure Virtual WAN support Announced, not GA GA
Azure Global Reach Limited to same geo-zone All regions
Max connections per circuit 10 100, depending on the circuit size (Mbps) – 20 for 50 Mbps, 100 for 10 Gbps+
Connections from different subscriptions No Yes
Max routes advertised Private peering: 4,000

Microsoft peering: 200

Private Peering: Up to 10,000

Microsoft peering: 200

I said “three tiers”, right? But there is also a third tier called Local which is very lightly documented. ExpressRoute Local is a subset of ExpressRoute Standard where:

  • The circuit can only connect to 1 or 2 Azure regions in the same metro as the POP or edge data centre. Therefore it is available in fewer locations than ExpressRoute Standard.
  • ExpressRoute Global Reach is not available.
  • It requires an unlimited data plan with at least 1 Gbps, coming in at ~25% of the price of a 1 Gbps Standard tier unlimited data plan.

Service Provider Types

There are three ways that a service provider can connect you to Azure using ExpressRoute, with two of them being:

  • Layer-2: A VLAN is stretched from your on-premises network to Azure
  • Layer-3: You connect to Azure over IP VPN or MPLS VPN. Your on-premises network connects either by BGP or a static default route.

There is a third option, called ExpressRoute Direct.

ExpressRoute Direct

A subset of the Microsoft POPs or edge data centres offer a third kind of connection for Azure ExpressRoute called ExpressRoute Direct. The features of this include:

  • Larger sizes: You can have sizes from 1 Gbps to 100 Gbps for massive data ingestion, for things like Cosmos DB or storage (HPC).
  • Physical Isolation: Some organisations will have a compliance reason to avoid connections to shared network equipment (the CEs and MSEE).
  • Granular control of circuit distribution: Based on business unit

This is a very specialised SKU that you must apply to use.

ExpressRoute FastPath

The normal flow of packets routing into Azure over ExpressRoute is:

  1. Enter Microsoft at the MSEE
  2. Travel via the ExpressRoute Virtual Network Gateway.
  3. If a route table exists, follow that route, for example, to a hub-based firewall.
  4. Route to the NIC of the virtual machine

There is a tiny latency penalty by routing through the Virtual Network Gateway. For a tiny percentage of customers, this latency may cause issues.

The concept of ExpressRoute Fast Path is that you can skip the hop of the virtual network gateway and route directly to the NICs of the virtual machines (in the same virtual network as the gateway).

To use this feature you must be using one of these gateway sizes:

  • Ultra Performance
  • ErGw3AZ

The following are not supported and will force traffic to route via the ExpressRoute Virtual Network Gateway:

  • There is a UDR on the GatewaySubnet
  • Virtual Network Peering is used. An alternative is to connect the otherwise-peered VNets directly to the circuit with their own VNet Gateway.
  • You use a Basic Load Balancer in front of the VMs; use a Standard tier Load Balancer.
  • You are attempting to connect to Private Endpoint.

ExpressRoute Global Reach

I think that ExpressRoute Global Reach is one of the more interesting features in ExpressRoute. You can have two or more offices, each with their own ExpressRoute (not Local tier) circuit to a local POP/edge data center, and enable Global Reach to allow:

  • The offices to connect to Azure/Microsoft cloud resources
  • Connect to each other over the Microsoft WAN instead of deploying a WAN

Note that ExpressRoute Standard will support connecting locations in the same geo-zone, and ExpressRoute Premium will support all geo-zones. Supported POPs are limited to a small subset of locations.

Encryption

Traffic over ExpressRoute is not encrypted and as Edward Snowden informed us, various countries are doing things to sniff traffic. If you wish to protect your traffic you will have to “bring your own key”.  We have a few options:

  • The aforementioned VPN over ExpressRoute, which is available now for Virtual Network Gateway and Azure Virtual WAN.
  • Implement a site-to-site VPN across ExpressRoute using a third-party virtual appliance hosted in the Azure VNet.
  • IPsec configured on each guest OS, limited to machines.
  • MACsec, a Layer-2 feature where you can implement your own encryption from your VE to the MSEE, encrypting all traffic, not just to/from VMs.

The MACsec key is stored securely in Azure Key Vault. From what I can see, MACsec is only available on ExpressRoute Direct. Microsoft claims that it does not cause a performance issue on their routers, but they do warn you to check your CE vendor guidance.

Multi-Cloud

Now you’ll see why I talked about Layer-2 and Layer-3. Depending on your service provider type and their connectivity to non-Microsoft clouds, if you have a circuit with the service provider (from your CEs to their CE facing PEs) that same circuit can be used to connect to Azure over ExpressRoute and to other clouds such as AWS or others. With BGP propagation, you could route from on-premises to/from either cloud, and your deployments in those clouds could route to each other.

Bidirectional Forwarding Detection (BFD)

The circuit is deployed as two connections, ideally connected to 2 CEs in your edge network. Failover is automated, but some will want failover to be as quick as possible. You can reduce the BGP keepalive and hold-time but this will be processor intensive on the network equipment.

A feature called BFD can detect link failure in a sub-second with low overhead. BFD is enabled on “newly” created ExpressRoute private peering interfaces on the MSEEs – you can reset the peering if required. If you want this feature then you need to enable it on your CEs – the service provider must also enable it on their PEs.

Monitoring

Azure Monitor provides a bunch of metrics for ExpressRoute that you can visualise or create alerts on.

Azure’s Connection Monitor is the Microsoft-offered solution for monitoring an ExpressRoute connection. The idea is that a Log Analytics agent (Windows or Linux) is deployed onto one or more always-on on-premises machines. A test is configured to run across the circuit measuring availability and performance.