The New Office – Construction Has Begun

It is day 2 of the construction of our new office. The carpenters arrived exactly at 9 am yesterday morning, bang-on time, to begin the assembly and construction of our new garden office – the global headquarters of Cloud Mechanix.

My only concern at the start was whether the site would be level enough for the build. The team warned us early on that the slope would make one side look higher up when it was level but we knew that and were OK with it. They quickly assembled the wooden frame that would eventually site on concrete blocks.

Once the 6x4m frame was built I was called out to pick the final position – allowing just over half a metre from the concrete garden wall so I can paint the outside to maintain the integrity of the wooden construction. Not long after the wooden frame was in position. Around lunchtime, the planks for the outside wall were being hammered into place. I’d say it took the 3 of them 45 minutes to get all the planks in place. Then the ceiling was put on.

The electricians were called in and asked to confirm how we wanted to do things after they surveyed the site. The one thing I was worried about was how the armoured electrical cable would run from the office to the fuse box at the front of the house – we need to power at least 2 computers + equipment and a 1.5 KW heater (please save comments on €13,000 hearing/aircon systems – the office is costing that much!). The ethernet line and electrical cable are entering on the back-right and will run along the garden wall under the capping (which you cannot see in the photos here). Then they will run underground to the front of the house. The ethernet line will enter the house at the same location as the phone/Internet and be just a meter from the broadband router. The electrical cable will cross the front of the house hidden in white trunking (maybe 2 cm wide) attached under the white soffit of the house – nicely camouflaged. The electrical cable will enter above the front door (still camouflaged) and then be under 1 metre from the fusebox – trunking from the front door, along an interior white doorframe – still camouflaged.

The electricians did the first fitting: 5 x dual power sockets, light switch, and 6 x dropdown lights. When leaving, they let me know that they would be back once the interior was finished by the carpenters to do the second fittings – sockets, lights, and the aforementioned cable installations.

The carpenters carried on, adding insulation and sheeting to the roof before the end of the day.

As you can see, there are two big windows, right where the front desks will be – one for me and one for my wife. Cloud Mechanix will have a great view overlooking about 40m of the garden!

Day two (today) began with drizzle. The team (down to two) were bang-on time again. They told me that they expect to require 3 days for the complete assembly and finishing of the wooden structure. Today, the plan was to get the doors and windows done, the studs for the wall insulation & interior installed, and the roof started. As you can see, the doors and windows were in place at lunchtime.

Exterior trims have been added, the slats for the roof (steel roof) are in place, and I can hear steel being cut as I type – they are installing the steel sheet roof that has a black tile effect. Once that is done, the slow bit (as they referred to it) will begin: the interior finishes, the lower trim and the step.

Azure Virtual WAN ARM – The Chicken & Egg Gateway ID Discombobulation

This post will explain how to deal with the gateway ID properties in the Azure Microsoft.Network/virtualhubs resource when using ARM templates.

Background

The Azure WAN Hub is capable of having 3 gateway sub-resources:

  • Point-to-site VPN: Microsoft.Network/p2sVpnGateways
  • VPN (site-to-site): Microsoft.Network/vpnGateways
  • ExpressRoute: Microsoft.Network/expressRouteGateways, which does not support diagnostic settings in the 2020-04-01 API

As you would expect, when you create these resources, you have to supply them with the resource ID of the Microsoft.Network/virtualhubs resource:

"virtualHub": {
  "id": "<<<<resource ID of the virtual hub>>>>"
},

What is a surprise is what happens in the Microsoft.Network/virtualhubs resource. After a gateway is associated, a property (type object, presumably for future-proofing) for the associated gateway type is added to the hub:

"vpnGateway": {
  "id": "<<<< Resource ID of Microsoft.Network/vpnGateways resource>>>>"
},
"expressRouteGateway": { 
 "id": "<<<< Resource ID of Microsoft.Network/p2sVpnGateways resource>>>>"
},
"p2SVpnGateway": { 
 "id": "<<<< Resource ID of Microsoft.Network/expressRouteGateways resource>>>>"
},

The surprising thing is what happens.

The Problem

There are 3 possible states in the hub when it comes to each gateway:

  1. The hub exists without a gateway: The above hub properties are not required.
  2. The gateways are being added: The above hub properties cannot be added because the gateway resource ID points to a resource that does not exist yet – the hub must exist and be configured before the gateway(s).
  3. The gateways exist: Any re-run of the ARM template (which might be common to update the hub route tables or configuration via DevOps) must include the above gateway properties in the hub resource with the correct resource IDs for the gateways.

And steps 2 and 3 are where the chicken and egg are in an ARM template. You must supply the gateway resource ID in the hub for all updates to the hub after a gateway is deployed, and you must not include the gateway resource ID in the hub when deploying the gateway. This would be easy to deal with if ARM would (finally) give us a “ifexists()” function but there is no sign of that. So we need a hack solution.

The Hack Solution

This one comes from the Well-Architected Framework/Cloud Adoption Framework, Enterprise-Scale Architecture. This way-too-complicated beastie shows how Microsoft’s people are dealing with the issue. The JSON for the Microsoft.Network/virtualhubs template contains these properties:

"properties": {
  "virtualWan": {
    "id": "[variables('vwanresourceid')]"
  },
  "addressPrefix": "[parameters('vHUB').addressPrefix]",
  "vpnGateway": "[if(not(empty(parameters('vHUB').vpnGateway)),parameters('vHUB').vpnGateway, json('null'))]"
}

The key for dealing with vpnGateway is the vHUB parameter, an object that contains a value called vpnGateway.

When they first run the deployment, the value of vHUB.vpngateway is set to {} or null in the parameters file, stored in GitHub. That means that when the hub is first run (and there is no VPN gateway), the if statement in the above snippet will pass json(‘null’) to the vpnGateway property. That is acceptable to the resource provider and the hub will deploy cleanly. Later on in the deployment, the VPN gateway will be created.

If you were to just re-run the hub template now, you will get an error about not being allowed to change the vpnGateway property in the hub resource. Behind the scenes it has been updated by the VPN gateway deployment. Every execution of the hub template must now include the resource ID of the VPN Gateway – that sucks, right? Now the hack really kicks in.

After the first deployment of the hub (and the VPN Gateway), you must open the resource group in the Azure Portal, enable viewing hidden items, open the VPN Gateway resource, go to properties, and document the resource ID.

Now, you need to open the parameters file for the hub. Edit the vHUB.vpnGateway property and set it to:

"vpnGateway": { 
 "id": "<<<< Resource ID of Microsoft.Network/vpnGateways resource>>>>"
},

Now you can cleanly re-run the hub template.

How Should It Work?

The best solution would be if the gateway ID properties were just documentation for Azure, properties that we humans cannot edit. But I suspect that the ability to configure these settings might have something to do with the newly announced NVA-in-hub preview. Otherwise, ARM needs to finally give us an ifexists() function – vote here now if you agree.

Azure Virtual WAN ARM – Secured Virtual Hub Azure Firewall

I have spent quite a few hours figuring out how to deploy Azure’s new Secured Virtual Hub, an extension of Azure Virtual WAN, deployed using ARM templates (JSON). A lot of the bits are either not documented or incorrectly documented. One of the frustrating bits to deploy was the Azure Firewall resource – and the online examples did not help.

The issue was that the 2 sources I could find did not include public IP addresses on the firewall:

  • The quick start for Secured Virtual Hub on docs.microsoft.com
  • The new Enterprise-Scale “well-architected” Framework, found in Cloud Adoption Framework

Digging to solve that uncovered:

  • The examples used quite an old API version, 2019-08-01, to deploy the Microsoft.Network/azureFirewalls resource.
  • There was no example of how to add a public IP address to the firewall in Secured Virtual Hub because it was not possible with that API – SVH is quite different from a VNet deployment because you do have direct access to the underlying hub virtual network.
  • Being an old API, we lose features such as SNAT for non-RFC1918 addresses (important in universities and public sector) and the newer custom & proxy DNS features.

In my digging, I did uncover that the ARM reference for the Azure Firewall was incorrect, but I did uncover a new, barely-documented property called hubIPAddresses; I knew this property was the key to solving the public IP address issue. So I thought about what was going on and how I was going to solve it.

I ended up doing what I would normally do if I did not have a quick start template to start with:

  1. Deploy the resource(s) by hand in the Azure Portal
  2. Observe the options – there was a slide control for the quantity of firewall public IP addresses
  3. Export the resulting template

And … there was the solution:

  1. There is a new, undocumented API version for the Azure Firewall resource: 2020-05-01
  2. There is a new object property called hubIPAddresses that contains an object sub-property called publicIps. You can set a string value called count to control how many public IP addresses that Azure will assign (on your behalf) to the firewall – you do not need to create the public IP address resources.
        "hubIPAddresses": {
          "publicIPs": {
            "count": "[parameters('firewallPublicIpQuantity')]",
          }
        }

Sorted!

The Office Installation Is About To Start

It’s a while (about 6 weeks) since I wrote that I am getting a new office built in the garden. Today seems like a good time to post an update.

There were 3 birch trees on the site where the office is being installed. I had the trees cut down (we have already replaced them with 4 apple trees and 2 more will be added next year) and that left the stumps. If I had been clever, I would have hired a mini-digger and dug around them. Maybe I would have also hired a tree grinder to chop up the revealed stumps. But no – instead we decided to dig them out by hand. Birch trees have interesting roots. The root ball is about 60-80 CM deep, and roots, sometimes 3x wider (or more) than the trunk, come out and up towards the surface. Those roots break out into a fine mesh that spreads out around the site, holding the soil firmly together, almost like concrete. I had a pick-axe, an axe, a shovel, a trowel, and my bare hands. The pick-axe was literally bouncing off the ground. The roots were like a phone book, and a sharp axe wasn’t cutting; it was breaking a layer or two and bounding too. I’ll keep this short – the final stump took over 8 hours to dig out by myself, and ended with me using my hands to scoop out dirt from under the root ball and roll the stump until the tap-root snapped.

Tip: hire the machinery, and don’t do it by hand.

The next thing to deal with was the base. Once the trees were removed, we realised that the site had a slant that we had never noticed before – about 27cm from left to right over 6m. I thought that would require a concrete base. So I put out some feelers to find a builder that would install the base. I got 4 names. Only 3 returned my contact attempts. Of the 3, only 2 bothered to show up – one of those was 2 days late! And of the 2, neither either bothered to give me a quote. I guess that builders are busy enough, even with all the COVID disruption that’s been going on. I called the cabin builders to get their opinion – and they said that we didn’t need a base. Their method of foundation installation would handle the minor slope. Perfect!

The electrician was the next thing. After my last post, I had lots of people telling me to go spend another 13,000-20,000 Euros on different kinds of heating systems – me arse! We didn’t even spend that on the house! I got an electrician quote – a guy who came recommended from the cabin builders. I’d tried reaching out to one other electrician but he was as useless at responding as the builders. The recommended guy is booked and will be doing the first (armored cable from the fusebox) and second fittings, as well as an Ethernet wire installation from our broadband router in the house to the office.

The last piece of the installation will be painting – that’s paid for with the installation and will be done afterward.

So when will all this happen? A lorry dropped off an “Ikea-like” load in our driveway a couple of hours ago. An installation team is due tomorrow at 9am to start the assembly. Hopefully we won’t have to wait too long for the painters and then we can get the alarm/security stuff installed, a new floor surface installed, and purchase some new office furniture.

Azure Virtual WAN Introducing A New Kind Of Route Table

In this post, I will quickly introduce you to a new kind of Route Table in Microsoft Azure that has been recently introduced by Azure Virtual WAN – and hence is included in the newly generally available Secured Virtual Hub.

The Old “Subnet” Route Table

This Route Table, which I will call “Subnet Route Table” (derived from the ARM name) is a simple resource that we associate with a subnet. It contains User-Defined Routes that force traffic to flow in desirable directions, typically when we use some kind of firewall appliance (Azure Firewall or third-party) or a third-party routing appliance. route The design is simple enough:

  • Name: A user-friendly name
  • Prefix: The CIDR you want to get to
  • Next Hop Type: What kind of “router” is the next hop, e.g. Virtual Network, Internet, or Virtual Appliance
  • Next Hop IP Address: Used when Next Hop Type is Virtual Appliance (any firewall or third-party router)

Azure Virtual WAN Hub

Microsoft introduced Azure Virtual WAN quite a while ago (by Cloud standards), but few still have heard of it, possibly because of how it was originally marketed as an SD-WAN solution compatible originally with just a few on-prem SD-WAN vendors (now a much bigger list). Today it supports IKEv1 and IKEv2 site-to-site VPN, point-to-site VPN, and ExpressRoute Standard (and higher). You might already be familiar with setting up a hub in a hub-and-spoke: you have to create the virtual network, the Route Table for inbound traffic, the firewall, etc. Azure Virtual WAN converts the hub into an appliance-like experience surfacing just two resources: the Virtual WAN (typically 1 global resource per organisation) and the hub (one per Azure region). Peering, routing, connectivity are all simplified.

A more recent change has been the Secured Virtual Hub, where Azure Firewall is a part of the Virtual WAN Hub; this was announced at Ignite and has just gone GA. Choosing the Secured Virtual Hub option adds security to the Virtual WAN Hub. Don’t worry, though, if you prefer a third-party firewall; the new routing model in Azure Virtual WAN Hub allows you to deploy your firewall into a dedicated spoke virtual network and route your isolated traffic through there.

The New Route Tables

There are two new kinds of route table added by the Virtual WAN Hub, or Virtual Hub, both of which are created in the Virtual Hub as sub-resources.

  • Virtual Wan Hub Route Table
  • Virtual WAN Route Table

Virtual WAN Hub Route Table

The Virtual Hub Hub Route Table affects traffic from the Virtual Hub to other locations.  A possible scenario is when you want to route traffic to a CIDR block of virtual network(s) through a third-party firewall (network virtual appliance/NVA):

AzureVirtualWanHubHubRouteTable

The routing rule setup here is similar to the Subnet Route Table, specifying where you want to get to (CIDR, resource ID, or service), the next hop, and a next hop IP address.

Virtual WAN Route Table

The Virtual WAN Route Table is created as a sub resource of the Virtual Hub but it has a different purpose. The Virtual Hub is assigned to connections and affects routing from the associated branch offices or virtual networks. Whoa, Finn! There is a lot of terminology in that sentence!

A connection is just that; it is a connection between the hub and another network. Each spoke connected directly to the hub has a connection to the hub – a Virtual WAN Route Table can be associated with each connection. A Virtual WAN Route Table can be associated with 1 virtual network connection, a subset of them, or all of them.

The term “branch offices” refers to sites connected by ExpressRoute, site-to-site VPN, or point-to-site VPN. Those sites also have connections that a Virtual WAN Route Table can be associated with.

This is a much more interesting form of route table. I haven’t had time to fully get under the covers here, but comparing ARM to the UI reveals two methodologies. The Azure Portal reveals one way of visualising routing that I must admit that I find difficult to scale in my mind. The ARM resource looks much more familiar to me, but until I get into a lab and fully test (which I hope I will find some hours to do soon), I cannot completely document.

Here are the basics of what I have gleaned from the documentation, which covers the Azure Portal method:

The linked documentation is heavy reading. I’m one of those people that needs to play with this stuff before writing too much in detail – I never trust the docs and, to be honest, this content is complicated, as you can see above.

Rethinking Firewall Management With Azure Firewall Manager

Microsoft has just announced the general availability a feature that I’ve been waiting for since I first learned about it last Autumn, called Azure Firewall Manager. Azure Firewall Manager allows you to centrally manage one or more Azure Firewall instances through a central, policy-driven, user interface. And it’s those policies, Azure Firewall Policies, that made me re-think Azure Firewall management a few months ago when I was writing my Cloud Mechanix course (running next ONLINE on July 30th) “Securing Azure Services & Data Through Azure Networking”.

Azure Firewall Policy

This is a new resource type that is generally available today. Azure Firewall Policy outsources the configuration and management of the firewall to a policy resource; that means that the usual settings in the Azure Firewall for things like rules and Threat Intelligence move from the firewall resource to a policy when a policy is associated with the firewall.

Policies can be created in a hierarchy. You can create a parent/global policy that will contain configurations and rules that will apply to all/a number of firewall instances. Then you create a child policy that inherits from the parent; note that rules changes in the parent instantly appear in the child. The child is associated with a firewall and applies configurations/rules from the parent policy and the child policy instantly to the firewall.

Problem

I’ve deployed and configured multiple customers where we have virtual data centers (VDCs, which are governed & secured hub and spoke architectures) across multiple regions. Creating rules configurations to allow flows from a spoke/service in one region to another spoke/service in another region is a royal pain in the tushie. Here’s the network flow (as I documented with routing here):

  1. Source device
  2. Outbound NSG rules in source spoke
  3. Firewall in source hub
  4. Firewall in destination hub
  5. Inbound NSG rules at destination spoke
  6. Destination device

There are potentially 4 sets of rules to configure for a simple service running on a single protocol/port. Today I configured Microsoft Identity Management for this scenario and there were dozens of protocol/port combinations across three spokes. The work took hours to complete – which I did in code and it provided a working result for the identity consulting team.

I minimise the work by controlling outbound flows in the local hub firewall, not in the NSG. So the NSGs do not control outbound flows at all. I could allow all via the firewall, even to other private networks, but that goes against the idea of compartmentalisation or micro-segmentation to combat modern network threats – so I need to configure both firewalls for a flow.

Solution

Re-think the firewall for a moment. Imagine you had one virtual firewall that spanned all of your Azure regional deployments. You can control all global flows with one configuration in that global virtual firewall. The global virtual firewall has instances in each Azure region. Any local flows can be configured just in that instance. That’s what Firewall Policy allows.

  • Parent Policy: Place all your global configurations in here. Some configurations will be company-wide, such as Threat Intelligence. Some rules, like allowing access to Microsoft URIs or Azure services (service tags) will be global too. And this is where you put the rules to allow flows between one regional deployment and another. This global management takes all your local Azure Firewall resources and treats them as a single security service.
  • Child Policies: A child policy will be created for each Azure Firewall instance. This policy will inherit the above from the parent applying the global configuration. Local rules, to allow north-south access to/from local services (Internet or on-prem) or east-west (spoke-to-spoke in the same regional deployment) will be configured here. RBAC can be enabled to allow local network admins to do their own thing, but unable to undo what the parent has done.

I haven’t had a chance to test Azure Firewall Policy out yet since the GA announcement, but I’m hoping that the third tier in rules (Rules Groups) made it from preview to GA. I do have groupings of rules collections based on buckets of priorities. This organisation would be awesome in my vision of Azure Firewall management.

Year 13 As A Microsoft Valuable Professional Begins

I found out today that I have been renewed as a Microsoft Azure MVP! This is my 13th consecutive year as an MVP, with previous expertises covering Hyper-V and System Center Configuration Manager. It’s an honour to continue to be part of this program.

The profile of how I am contributing has changed a lot. In my early days, I wrote about everything I saw in my RSS feeds – and I mean everything. I was single with no responsibilities after work, and often living in a hotel with nothing better to do. I got involved in writing books (remember those?). I wrote my own whitepapers – a full week or two of work. Today, Microsoft does a much better (not perfect) job at documenting things. There are lots of people in the community sharing content today so things I might have written about are being done elsewhere and why create redundant content? ]Today, I have a wife and young family and they are my priority in my off-hours. So I have changed how I contribute to the community.

I prefer events where I can fly in out the same day, or at least limit my away time to a quick overnight visit. But that’s all ended for us lately, so the online medium has taken over, which makes me much more available. I haven’t written a tech book in years – my last book (on WS2012 Hyper-V) was when the content was relatively static; today, a book on any tech is out of date before you save your markdown file. I pick and choose my blog topics – I spend a LOT of time doing Azure networking/security and I’ve always skewed my blogging to what I work on. I’ve found even senior consultants/architects don’t know this stuff well so my recent posts have gone into the level-400 stuff on routing and security design/implementation that is usually misunderstood. A lot of my community contribution isn’t obvious.

The MVP program provides MVPs with direct contact with the program managers of the products in their expertise area, and other areas. I’ve taken advantage of that since my 2nd year in the program. There are things in the Hyper-V/clustering/storage world that I can point & say “I provided feedback to make that happen”. And that continues in Azure; like a lot of MVPs, I give feedback and end up in 1:1 email/Teams conversations with the relevant PMs. There are recent things that have launched (some still coming) that I have helped shape. That’s one of the cool parts of being a member of the program, being able to take my observations from work or feedback from customers/community members, and bring it to the product group to improve things for all of us. I think that’s where I’ve been most busy this last year.

There are cool things coming … some are there already but you might not know it! … and that’ll give us lots of things to talk about 🙂

July 30th – Online Training: Securing Azure Services & Data Through Azure Networking

My company, Cloud Mechanix, will be delivering a one-day online training course on July 30th called Securing Azure Services & Data Through Azure Networking.

I set up Cloud Mechanix a few years ago to deliver custom-written training in Azure. Year 1, 2018, was good; I ran two classes in London, and classes in Amsterdam and Frankfurt. I started a new job in 2019 and didn’t feel like I could put the time into Cloud Mechanix that paying customers deserve. With 2020, I felt like I could put together a new course on Azure networking/security based on my experience. I was approached by a company in Belgium to run a course with them; they would use their customer database to promote the class and I would teach. Unfortunately, their promotion failed and the course only sold a few seats. Timing-wise, the class was meant to happen at the start of the COVID-19 pandemic in Europe so cancellation wasn’t a completely bad thing. Then a few months ago, the European Cloud Conference asked if I would do a master class on Azure. I had most of the content I wanted for a class and this seemed like a good place to kick the class off before I run it again with Cloud Mechanix. That class was advertised by the European Cloud Conference as a free class for up to 15 attendees – over 400 applied! I was convinced – bring the class to the public.

So that’s what I am doing. Later today, I’m presenting a teeny-tiny snippet of the PowerPoint deck at E2EVC “Madrid” Online. On June 19th, I will be doing the Azure Masterclass that is sponsored by the European Cloud Conference. And as of now, my class for July 30th is on sale.

Security is always number 1 or 2 in any survey on the fears of cloud computing. Networking in The Cloud is very different from traditional physical networking … but in some ways, it is quite similar. The goal of this workshop is to teach you how to secure your services and data in Microsoft Azure using techniques and designs that are advocated by Microsoft Azure. Don’t fall into the trap of thinking that networking means just virtual machines; Azure networking plays a big (and getting bigger) role in offering security and compliance with platform and data services in The Cloud.

This online class takes you all the way back to the basics of Azure networking so you really understand the “wiring” of a secure network in the cloud. Only with that understanding do you understand that small is big. The topics covered in this class will secure small/mid businesses, platform deployments that require regulatory compliance, and large enterprises:

  • The Microsoft global network
  • Availability & SLA
  • Virtual network basics
  • Virtual network adapters
  • Peering
  • Service endpoints
  • Public IP Addresses
  • VNet gateways: VPN & ExpressRoute
  • Network Security Groups
  • Application Firewall
  • Route Tables
  • Platform services & data
  • Private Link & Private Endpoint
  • Third-Party Firewalls
  • Azure Firewall
  • Monitoring
  • Troubleshooting
  • Security management
  • Micro-Segmentation
  • Architectures

The class is online only – so no matter where you are you will be able to participate. There are no travel restrictions, no travel or accommodation costs, and all you will need is the ability to get online. I will be presenting using Microsoft Teams – you won’t need a license because of the licensing that I will be using. The ticket cost is low – €229 per person, or €209 for 2+ tickets.

I hope to see some of you there on July 30th, starting at 09:00 UK/Irish time (+5 from EST and -1 from CET)!

Connecting Azure Hub-And-Spoke Architectures Together

In this post, I will explain how you can connect multiple Azure hub-and-spoke (virtual data centre) deployments together using Azure networking, even across different Azure regions.

There is a lot to know here so here is some recommended reading that I previously published:

If you are using Azure Virtual WAN Hub then some stuff will be different and that scenario is not covered fully here – Azure Virtual WAN Hub has a preview (today) feature for Any-to-Any routing.

The Scenario

In this case, there are two hub-and-spoke deployments:

  • Blue: Multiple virtual networks covered by the CIDR of 10.1.0.0/16
  • Green: Another set of multiple virtual networks covered by the CIDR of 10.2.0.0/16

I’m being strategic with the addressing of each hub-and-spoke deployment, ensuring that a single CIDR will include the hub and all spokes of a single deployment – this will come in handy when we look at User-Defined Routes.

Either of these hub-and-spoke deployments could be in the same region or even in different Azure regions. It is desired that if:

  • Any spoke wishes to talk to another spoke it will route through the local firewall in the local hub.
  • All traffic coming into a spoke from an outside source, such as the other hub-and-spoke, must route through the local firewall in the local hub.

That would mean that Spoke 1 must route through Hub 1 and then Hub 2 to talk to Spoke 4. The firewall can be a third-party appliance or the Azure Firewall.

Core Routing

Each subnet in each spoke needs a route to the outside world (0.0.0.0/0) via the local firewall. For example:

  • The Blue firewall backend/private IP address is 10.1.0.132
  • A Route Table for each subnet is created in the Blue deployment and has a route to 0.0.0.0/0 via a virtual appliance with an IP address of 10.1.0.132
  • The Greenfirewall backend/private IP address is 10.2.0.132
  • A Route Table for each subnet is created in the Green deployment and has a route to 0.0.0.0/0 via a virtual appliance with an IP address of 10.2.0.132

Note: Some network-connected PaaS services, e.g. API Management or SQL Managed Instance, require additional routes to the “control plane” that will bypass the local firewall.

Site-to-Site VPN

In this scenario, the organisation is connecting on-premises networks to 1 or more of the hub-and-spoke deployments with a site-to-site VPN connection. That connection goes to the hub of Blue and to Green hubs.

To connect Blue and Green you will need to configure VNet Peering, which can work inside a region or across regions (using Microsoft’s low latency WAN, the second-largest private WAN on the planet). Each end of peering needs the following settings (the names of the settings change so I’m not checking their exact naming):

  • Enabled: Yes
  • Allow Transit: Yes
  • Use Remote Gateway: No
  • Allow Gateway Sharing: No

Let’s go back and do some routing theory!

That peering connection will add a hidden Default (“system”) route to each subnet in the hub subnets:

  • Blue hub subnets: A route to 10.2.0.0/24
  • Green hub subnets: A route to 10.1.0.0/24

Now imagine you are a packet in Spoke 1 trying to get to Spoke 4. You’re sent to the firewall in Blue Hub 1. The firewall lets the traffic out (if a rule allows it) and now the packet sits in the egress/frontend/firewall subnet and is trying to find a route to 10.2.2.0/24. The peering-created Default route covers 10.2.0.0/24 but not the subnet for Spoke 4. So that means the default route to 0.0.0.0/0 (Internet) will be used and the packet is lost.

To fix this you will need to add a Route Table to the egress/frontend/firewall subnet in each hub:

  • Blue firewall subnet Route Table: 10.2.0.0/16 via virtual appliance 10.2.0.132
  • Red firewall subnet Route Table: 10.1.0.0/16 via virtual appliance 10.1.0.132

Thanks to my clever addressing of each hub-and-spoke, a single route will cover all packets leaving Blue and trying to get to any spoke in Red and vice-versa.

ExpressRoute

Now the customer has decided to use ExpressRoute to connect to Azure – Sweet! But guess what – you don’t need 1 expensive circuit to each hub-and-spoke.

You can share a single circuit across multiple ExpressRoute gateways:

  • ExpressRoute Standard: Up to 10 simultaneous connections to Virtual Network Gateways in 1+ regions in the same geopolitical region.
  • ExpressRoute Premium: Up to 100 simultaneous connections to Virtual Network Gateways in 1+ regions in any geopolitical region.

FYI, ExpressRoute connections to the Azure Virtual WAN Hub must be of the Premium SKU.

ExpressRoute is powered by BGP. All the on-premises routes that are advertised propagate through the ISP to the Microsoft edge router (“meet-me”) in the edge data centre. For example, if I want an ExpressRoute circuit to Azure West Europe (Middenmeer, Netherlands – not Amsterdam) I will probably (not always) get a circuit to the POP or edge data centre in Amsterdam. That gets me a physical low-latency connection onto the Microsoft WAN – and my BGP routes get to the meet-me router in Amsterdam. Now I can route to locations on that WAN. If I connect a VNet Gateway to that circuit to Blue in Azure West Europe, then my BGP routes will propagate from the meet-me router to the GatewaySubnet in the Blue hub, and then on to my firewall subnet.

BGP propagation is disabled in the spoke Route Tables to ensure all outbound flows go through the local firewall.

But that is not the extent of things! The hub-and-spoke peering connections allow Gateway Sharing from the hub and Use Remote Gateway from the spoke. With that configuration, BGP routes to the spoke get propagated to the GatewaySubnet in the hub, then to the meet-me router, through the ISP and then to the on-premises network. This is what our solution is based on.

Let’s imagine that the Green deployment is in North Europe (Dublin, Ireland). I could get a second ExpressRoute connection but:

  • That will add cost
  • Not give me the clever solution that I want – but I could work around that with ExpressRoute Global Reach

I’m going to keep this simple – by the way, if I wanted Green to be in a different geopolitical region such as East US 2 then I could use ExpressRoute Premium to make this work.

In the Green hub, the Virtual Network Gateway will connect to the existing ExpressRoute circuit – no more money to the ISP! That means Green will connect to the same meet-me router as Blue. The on-premises routes will get into Green the exact same way as with Blue. And the routes to the Green spokes will also propagate down to on-premises via the meet-me router. That meet-me router knows all about the subnets in Blue and Green. And guess what BGP routers do? They propagate – so, the routes to all of the Blue subnets propagate to Green and vice-versa with the next hop (after the Virtual Network Gateway) being the meet-me router. There are no Route Tables or peering required in the hubs – it just works!

Now the path from Blue Spoke 1 to Green Spoke 4 is Blue Hub Firewall, Blue Virtual Network Gateway, <the Microsoft WAN>, Microsoft (meet-me) Router, <the Microsoft WAN>, Green Virtual Network Gateway, Green Hub Firewall, Green Spoke 4.

There are ways to make this scenario more interesting. Let’s say I have an office in London and I want to use Microsoft Azure. Some stuff will reside in UK South for compliance or performance reasons. But UK South is not a “hero region” as Microsoft calls them. There might be more advanced features that I want to use that are only in West Europe. I could use two ExpressRoute circuits, one to UK South and one to West Europe. Or I could set up a single circuit to London to get me onto the Microsoft WAN and connected this circuit to both of my deployments in UK South and West Europe. I have a quicker route going Office > ISP > London edge data center > Azure West Europe than from Office > ISP > Amsterdam edge data center > Azure West Europe because I have reduced the latency between me and West Europe by reducing the length of the ISP circuit and using the more-direct Microsoft WAN. Just like with Azure Front Door, you want to get onto the Microsoft WAN as quickly as possible and let it get you to your destination as quickly as possible.