In this article, I want to explain how important BGP is in Azure networking, even if you do not actually use BGP for routing, and the major role it plays in hub-and-spoke architectures and deployments with a firewall.
What is BGP?
I was never the network guy in an on-premises deployment. Those 3 letters, BGP, were something someone else worried about. But in Azure, the server admin becomes a network admin. Most of my work in Azure is networking now. And that means that the Border Gateway Protocol (BGP) is important to me now.
BGP is a means of propagating routes around a network. It’s a form of advertising or propagation that spreads routes to one or more destinations one hop at a time. If you think about it, BGP is like word-of-mouth.
A network, Subnet A, is a destination. Subnet A advertises a route to itself to a neighbour network, Subnet B. Subnet B advertises to its neighbours, including Subnet C, that it knows how to get to the original subnet, Subnet A. And the propagation continues. A subnet at the far end of the LAN/WAN, Subnet D, knows that there is another subnet far away called Subnet A and that the path to Subnet A is back via the propagating neighbour, Subnet C. Subnet C will then forward the traffic to Subnet B, which in turn sends the traffic to the destination subnet, Subnet A.
Azure and BGP
Whether you use BGP in your on-premises network or not, there will be a pretty high percentage chance that you will use BGP in Azure virtual networking – we’ll get to that in a few moments.
If you create a site-to-site VPN connection, you have the option to integrate your on-premises BGP routing with your Azure virtual network(s). If you use ExpressRoute, you must use BGP. In both cases, BGP routes are propagated from on-premises, informing your Azure virtual network gateway of all the on-premises networks that it can route to over that connection.
But BGP Is Used Without BGP
Let’s say that you are deploying a site-to-site VPN connection to Azure and that you do not use BGP in your configuration. Instead, you create a Local Network Gateway in Azure to define your on-premises networks. The virtual network gateway will load those networks from the Local Network Gateway and know to route across the associated VPN tunnel to get to those destinations.
And here’s where things get interesting. Those routes must get advertised around the virtual network.
If a virtual machine in the virtual network needs to talk to on-premises, it needs to know that the route to that on-premises subnet is via the VNet Gateway in the gateway subnet. So, the route gets propagated out from the gateway subnet.
Let’s scale that situation out a bit to a hub & spoke architecture. We have a site-to-site connection with or without BGP being used. The routes to on-premises are in the VNet Gateway and are propagated out to the subnets in the hub VNet that contains the VNet Gateway. And in turn, the routes are advertised to peered virtual networks (spokes) and their subnets. Now a resource on a subnet in a spoke virtual network has a route to an on-premises virtual network – across the peering connection and to the virtual network gateway.
Note: in this scenario, the hub is sharing the VNet gateway via peering, and the spoke is configured in peering to use the remote VNet gateway.
Routing is always a 2-way street. If routes only went one way, then a client could talk to a server, but the server would not be able to talk to the client.
If we have BGP enabled VPN or ExpressRoute, then Azure will propagate routes for the spoke subnets back down through peering and to the VNet Gateway. The VNet Gateway will then propagate those routes back to on-premises.
If you do not have BGP VPN (you are statically setting up on-premises routes in the Local Network Gateway) then you will have to add the address space of each spoke subnet to the on-premises VPN appliance(s) so that they know to route via the tunnel to get to the spokes. The simple way to do that is to plan your Azure networking in advance and have a single supernet (a /16, for example) instead of a long list of smaller subnets (/24s, for example) to configure.
Control & Security
Let’s say that you want to add a firewall to your hub. You want to use this firewall to isolate everything outside of Azure from your hub and spoke architecture, including the on-premises networks. You’ve done some research and found that you need to add a route table and a user-defined route to your hub and spoke subnets, instructing them that the route to on-premises is through the VNet Gateway.
Now you need to do some reading – you need to learn (1) how Azure routing really works (not how you think it works) and (2) how to troubleshoot Azure routing. FYI, I’ve been living in this world non-stop for the last 10 months.
What you will probably have done is configured your spokes with a route to 0.0.0.0/0 via the internal/backend IP address of the firewall. You are assuming that will send all traffic to anywhere via the Firewall. Under the covers, though, routes to on-premises are still propagating from the VNet Gateway to all the subnets in your hub and spoke architecture. If on-premises was 192.168.1.0/24 and your spoke machine wanted to route to on-premises, then the Azure network fabric will compare the destination with the routes that it has in a hidden route table – the only place you can see this is in Effective Routes in a VM NIC Azure resource. You have a UDR for 0.0.0.0/0 via the firewall. That’s a 0-bit match for any destinations in 192.168.1.0/24. If that was the only route in the subnet, then that route would be taken. But we are sending a packet to 192.168.1.x and that is a 24-bit match with the propagated route to 192.1681.0/24. And that’s why the response from the spoke resource will bypass the firewall and go straight to the VNet Gateway (via peering) to get to on-premises. That is not what you expected or wanted!
Note: the eagle-eyed person that understands routing will know that there will be other routes in the subnet, but they are irrelevant in this case and will confuse the explanation.
The following works even if you do not use BGP with a site-to-site VPN!
To solve this problem, we can stop propagation – we can edit the route table resources in the internal Azure subnets (or pre-do this in JSON) and disable BGP route propagation. The result of this is that the routes that the VNet Gateway were pushing out to other subnets will stop being propagated. Now if we viewed the effective routes for a spoke subnet, we’d only see a route to the firewall and the firewall is now responsible for forwarding traffic to on-premises networks to the VNet Gateway.
It is important to understand that this disabling of propagation affects the propagation only in 1 direction. Routes from the VNet Gateway will not be propagated to subnets with propagation disabled. However, ALL subnets will still propagate routes to themselves back to the VNet Gateway – we need on-premises to know that the route to these Azure subnets is still via the Gateway.
More work will be required to get the Gateway Subnet to route via the firewall, but that’s a whole other topic! We’re sticking to BGP and propagation here.
The Firewall and BGP Propagation
Let’s make a mistake, shall we? It will be useful to get a better understanding of the features. We shall add a route table to the firewall subnet and disable BGP route propagation. Now the resource in the spoke subnet wants to send something to an on-premises network. The local subnet route table instructs it to send all traffic to external destinations (0.0.0.0/0) via the firewall. The packets hit the firewall. The firewall tries to send that traffic out and … it has only one route (a simplification) which is to send 0.0.0.0/0 to Internet.
By disabling BGP propagation on the firewall subnet, the firewall no longer knows that the route to on-premises networks is via the virtual network gateway. This is one of those scenarios where people claim that their firewall isn’t logging traffic or flows – in reality, the traffic is bypassing the firewall because they haven’t managed their routing.
The firewall must know that the on-premises networks (a) exist and (b) are routes to via the VNet Gateway. Therefore, BGP propagation must be left enabled on the firewall subnet (the frontend one, if you have a split frontend/backend firewall subnet design).
Not Just Firewalls!
I’m not covering it here, but there are architectures where there might be other subnets that must bypass the firewall to get back to on-premises. In those cases, those subnets must also have BGP propagation left enabled – they must know that the on-premises networks exist and that they should route via the VNet Gateway.
14 thoughts on “BGP with Microsoft Azure Virtual Networks & Firewalls”
Great write up.
Thanks for the great article Aidan.
It is not well explained in Microsoft documentation, so this has saved me a lot of time!
Awesome write up. Something that is not clearly documented elsewhere. Unfortunately I didn’t find it until after I had investigated it for a good while and come to the same conclusion but very reassuring to read your findings! You always manage to write up real life, potentially complex scenarios in an easy to understand way.
This was very helpful!
My situation is ExpressRoute to onprem. I tried creating the 0.0.0.0/0 route in a route table like you said from a workload subnet in azure to go to the azure firewall. I then created another route table to route to the azure workload subnet via the firewall and associated that with the gatewaysubnet of the ExpressRoute gateway.
I’m so confused as to why traffic is not flowing between on prem and azure workload subnet when I use these route tables and the azure firewall.
If I remove the route tables everything just works as expected between the workload subnet and azure via ExpressRoute.
Your article made me start thinking but I’m still confused. Any idea how to accomplish onprem to and from azure sinners via ExpressRoute with azure firewall in the middle?
Any help would be much appreciated.
Have you disabled BGP route propagation on the spoke subnet? Have you created rules in the firewall to allow the traffic? What I have described does work – it’s what is intended by the product group and what I (and others) have deployed for many customers.
Hi! first of all, great article! I am working in a project very similar to this, and you have confirmed a lot of things that I though but wasn’t totally sure.
You said “More work will be required to get the Gateway Subnet to route via the firewall, but that’s a whole other topic!”
Have you posted something about this?
I want to route all the traffic coming from on premises trough a Firewall… Is it enough just adding a UDR to the gateway subnet pointing to the firewall?
note* I have BGP enable on the VNG.
On the gateway subnet route table, create a UDR for each spoke VNet via the firewall. This is required to override the default (system) peering route that will go straight to the spoke, bypassing the firewall.
Excellent article and very well explained!
“More work will be required to get the Gateway Subnet to route via the firewall, but that’s a whole other topic!” – you should consider writing your article on that topic. It will be very popular 🙂
This is great, finally found an article detailing what I want to achieve. Only problem is this article is very high level and doesn’t actually discuss the configuration to achieve it. Nothing I’ve tried works, there’s no way to stop BGP propagation in only one direction, it’s either BGP propagation in both directions or nothing at all. Is there more specific configuration documentation available?
This is all great stuff and amazing article Aidan!
I wanted to share a scenario that might help others since I spend countless hours by myself and MS support figuring out why it doesn’t work.
I was trying to route all traffic through the Azure firewall.
Setup included BGP and Vnet-to-Vnet IPSec connections, instead of Vnet peering.
Microsoft ended up responding that routing traffic to the firewall in a Vnet-to-Vnet IPSec is not supported.
This is because all routing has to go through the GatewaySubnet and there is a loop between the GatewaySubnet of the Hub Vnet and the Firewall.
Ended up reverting to Vnet peering instead and configured route tables as Aidan suggested.