Tag: Network Virtual Appliance

How to Troubleshoot Azure Routing?

This post will explain how routing works in Microsoft Azure, and how to troubleshoot your routing issues with Route Tables, BGP, and User-Defined Routes in your virtual network (VNet) subnets and virtual (firewall) appliances/Azure Firewall.

Software-Defined Networking

Right now, you need to forget VLANs, and how routers, bridges, routing switches, and all that crap works in the physical network. Some theory is good, but the practice … that dies here.

Azure networking is software-defined (VXLAN). When a VM sends a packet out to the network, the Azure Fabric takes over as soon as the packet hits the virtual NIC. That same concept extends to any virtual network-capable Azure service. From your point of view, a memory copy happens from source NIC to destination NIC. Yes; under the covers there is an Azure backbone with a “more physical” implementation but that is irrelevant because you have no influence over it.

So always keep this in mind: network transport in Azure is basically a memory copy. We can, however, influence the routing of that memory copy by adding hops to it.

Understand the Basics

When you create a VNet, it will have 1 or more subnets. By default, each subnet will have system routes. The first ones are simple, and I’ll make it even more simple:

  • Route directly via the default gateway to the destination if it’s in the same supernet, e.g. 10.0.0.0/8
  • Route directly to Internet if it’s in 0.0.0.0/0

By the way, the only way to see system routes is to open a NIC in the subnet, and click Effective Routes under Support & Troubleshooting. I have asked that this is revealed in a subnet – not all VNet-connected services have NICs!

And also, by the way, you cannot ping the subnet default gateway because it is not an appliance; it is a software-defined function that is there to keep the guest OS sane … and probably for us too 😊

When you peer a VNet with another VNet, you do a few things, including:

  • Instructing VXLAN to extend the plumbing of between the peered VNets
  • Extending the “VirtualNetwork” NSG rule security tag to include the peered neighbour
  • Create a new system route for peering.

The result is that VMs in VNet1 will send packets directly to VMs in VNet2 as if they were in the same VNet.

When you create a VNet gateway (let’s leave BGP for later) and create a load network connection, you create another (set of) system routes for the virtual network gateway. The local address space(s) will be added as destinations that are tunnelled via the gateway. The result is that packets to/from the on-prem network will route directly through the gateway … even across a peered connection if you have set up the hub/spoke peering connections correctly.

Let’s add BGP to the mix. If I enable ExpressRoute or a BGP-VPN, then my on-prem network will advertise routes to my gateway. These routes will be added to my existing subnets in the gateway’s VNet. The result is that the VNet is told to route to those advertised destinations via the gateway (VPN or ExpressRoute).

If I have peered the gateway’s VNet with other VNets, the default behaviour is that the BGP routes will propagate out. That means that the peered VNets learn about the on-premises destinations that have been advertised to the gateway, and thus know to route to those destinations via the gateway.

And let’s stop there for a moment.

Route Priority

We now have 2 kinds of route in play – there will be a third. Let’s say there is a system route for 172.16.0.0/16 that routes to virtual network. In other words, just “find the destination in this VNet”. Now, let’s say BGP advertises a route from on-premises through the gateway that is also for 172.16.0.0/16.

We have two routes for the 172.16.0.0/16 destination:

  • System
  • BGP

Azure looks at routes that clash like above and deactivates one of them. Azure always ranks BGP above System. So, in our case, the System route for 172.16.0.0/16 will be deactivated and no longer used. The BGP route for 172.16.0.0/16 via the VNet gateway will remain active and will be used.

Specificity

Try saying that word 5 times in a row after 5 drinks!

The most specific route will be chosen. In other words, the route with the best match for your destination is selected by the Azure fabric. Let’s say that I have two active routes:

  1. 16.0.0/16 via X
  2. 16.1.0/24 via Y

Now, let’s say that I want to send a packet to 172.16.1.4. Which route will be chosen? Route A is a 16 bit match (172.16.*.*). Route B is a 24 bit match (172.16.1.*). Route B is a closer match so it is chosen.

Now add a scenario where you want to send a packet to 172.16.2.4. At this point, the only match is Route A. Route B is not a match at all.

This helps explain an interesting thing that can happen in Azure routing. If you create a generic rule for the 0.0.0.0/0 destination it will only impact routing to destinations outside of the virtual network – assuming you are using the private address spaces in your VNet. The subnets have system routes for the 3 private address spaces which will be more specific than 0.0.0.0:

  1. 168.0.0/16
  2. 16.0.0/12
  3. 0.0.0/8
  4. 0.0.0/0

If your VNet address space is 10.1.0.0/16 and you are trying to send a packet from subnet 1 (10.1.1.0/24) to subnet 2 (10.1.2.0/24), then the generic Route D will always be less specific than the system route, Route C.

Route Tables

A route table resource allows us to manage the routing of a subnet. Good practice is that if you need to manage routing then:

  • Create a route table for the subnet
  • Name the route table after the VNet/subnet
  • Only use a route table with 1 subnet

The first thing to know about route tables is that you can control BGP propagation with them. This is especially useful when:

  • You have peered virtual networks using a hub gateway
  • You want to control how packets get to that gateway and the destination.

The default is that BGP propagation is allowed over a peering connection to the spoke. In the route table (Settings > Configuration) you can disable this propagation so the BGP routes are never copied from the hub network (with the VNet gateway) to the peered spoke VNet’s subnets.

The second thing about route tables is that they allow us to create user-defined routes (UDRs).

User-Defined Routes

You can control the flow of packets using user-defined routes. Note that UDRs outrank BGP routes and System Routes:

  1. UDR
  2. BGP routes
  3. System routes

If I have a system or BGO route to get to 192.168.1.0/24 via some unwanted path, I can add a UDR to 192.168.1.0/24 via the desired path. If the two routes are identical destination matches, then my UDR will be active and the BGP/system route will be deactivated.

Troubleshooting Tools

The traditional tool you might have used is TRACERT. I’m sorry, it has some use, but it’s really not much more than PING. In the software defined world, the default gateway isn’t a device with a hop, the peering connection doesn’t have a hop, and TRACERT is not as useful as it would have been on-premises.

The first thing you need is the above knowledge. That really helps with everything else.

Next, make sure your NSGs aren’t the problem, not your routing!

Next is the NIC, if you are dealing with virtual machines. Go to Effective Routes and look at what is listed, what is active and what is not.

Network Watcher has a couple of tools you should also look at:

  • Next Hop: This is a pretty simple tool that tells you the next “appliance” that will process packets on the journey to your destination, based on the actual routing discovered.
  • Connection Troubleshoot: You can send a packet from a source (VM NIC or Application Gateway) to a certain destination. The results will map the path taken and the result.

The tools won’t tell you why a routing plan failed, but with the above information, you can troubleshoot a (desired) network path.

Why Choose the Azure Firewall over a Virtual Firewall Appliance?

In this post, I will explain why you should choose Azure Firewall over third-party firewall network virtual appliances (NVAs) from the likes of Cisco, Palo Alto, Check Point, and so on.

Microsoft’s Opinion

Microsoft has a partner-friendly line on Azure Firewall versus third-parties. Microsoft says that third-party solutions offer more than Azure Firewall. If you want you can use them side-by-side.

Now that’s out of the way, let me be blunt … like I’d be anything else! 😊

The NVA Promise

At their base, a firewall blocks or allows TCP/UDP/etc and does NAT. Some firewalls offer a “security bundle” of extra features such as:

  • Malware scanning based on network patterns
  • Download scanning, including zero-days (detonation chamber)
  • Browser URL logging & filtering

But those cool things either make no sense in Azure or are just not available from the NVA vendors in their cloud appliances. So what you are left with is central logging and filtering.

Documentation

With the exception of Palo Alto (their whitepaper for Azure is very good – not perfect) and maybe Check Point, the vendors have pretty awful documentation. I’ve been reading a certain data centre mainstay’s documents this week and they are incomplete and rubbish.

Understanding of Azure

It’s quite clear that some of the vendors are clueless about The Cloud and/or Azure. Every single vendor has written docs about deploying everything into a single VNet – if you can afford NVAs then you are not putting all your VMs into a single VNet (see hub & spoke VNet peering). Some have never heard of availability zones – if you can afford NVAs then you want as high an SLA as you can get. Most do not offer scale-out (active/active clusters) – so a single VM becomes your bottleneck on VM performance (3000 Mbps in a D3_v2). Some don’t even support highly available firewall clusters – so a single VM becomes the single point of failure in your entire cloud network! And their lack of documentation or understanding of VNet peering or route tables in a large cloud deployment is laughable.

The Comparison

So, what I’m getting at is that the third-party NVAs suck. Azure Firewall isn’t perfect either, but it’s a true cloud platform service and it is improving fast – just last night Microsoft announced Threat Intelligence-Based Filtering and Service Tags Filtering (this appeared recently). I know more things are on the way too 😊

Here is my breakdown of how Azure Firewall stacks up against firewall NVAs:

Azure Firewall NVA
Deployment Platform Linux VM + Software
Licensing Consumption: instance + GB Linux VM + Software
Scaling Automatic Add VMs + Software
Ownership Set & monitor Manage VM / OS / Software
Layer -7 Logging & filtering Potentially* deep inspection
Networking 1 subnet & PIP 1+ subnets & 1 PIP
Complexity Simple Difficult

I know: you laugh when you hear “Microsoft” and “Firewall” in the same sentence. You think of ISA Server. Azure Firewall is different. This is baked into the fabric of Azure, the strategic future of Microsoft. It is already rapidly improving, and it does more than the third parties.

Heck, what does the third-party offer compared to NSGs? NSGs filter TCP/UDP, they can log to a storage account, you can centrally log using Event Hubs, and does advanced reporting/analysis using NSG Flo Logs with Azure Monitor Logs (Log Analytics). Azure Firewall takes that another step with a hub deployment, an understanding of HTTP/S, and is now using machine learning for dynamic threat prevention!

My Opinion

Some people will always prefer a non-Microsoft firewall. But my counter would be, what are you getting that is superior – really? With Azure Firewall, I create a firewall, set my rules, configure my logging, and I’m done. Azure Firewall scales and it is highly available. Logging can be done to storage accounts, event hubs (SIEM), and Azure Monitor Logs. And here’s the best bit … it is SIMPLE to deploy and there is almost no cost of ownership. Compare that to some of the HACK solutions from the NVA vendors and you’d laugh.

The Azure Firewall was designed for The Cloud. It was designed for the way that Azure works. And it was designed for how we should use The Cloud … at scale. And that scale isn’t just about Mbps, but in terms of backend services and networks. From what I have seen so far, the same cannot be said for firewall NVAs. For me, the decision is easy: Azure Firewall. Every time.