Tag: Private Link

Enable FQDN-Based Network Rules In Azure Firewall

In this post, I will discuss how the DNS features, DNS Servers and DNS Proxy, can be used to enable FQDN-based rules in the Azure Firewall.

Can’t Azure Firewall Already Do FQDN-based Rules?

Yes – and no. One of the rules types is Application Rules, which control outbound access to services which can be based on a URI (a DNS name) for HTTP/S and SQL (including SQL Server, Azure SQL, etc) services. But this feature is not much use if:

  • You have some service in one of your VNets that needs to make an outbound connection on TCP 25 to something like smtp.office365.com.
  • You need to allow an inbound connection to an FQDN, such as a platform resource that is network-connected using Private Endpoint.

FQDN-Based Network Rules

Network rules allow you to control flows in/out from source to destinations on a particular protocol and source/destination port. Originally this was, and out of the box this is, done using IPv4 addresses/CIDRs. But what if I need to have some network-connected service reach out to smtp.office365.com to send an email? What IP address is that? Well, it’s lots of addresses:

nslookup smtp.office365.com

Non-authoritative answer:
Name: DUB-efz.ms-acdc.office.com
Addresses: 2603:1026:c02:301e::2
2603:1026:c02:2860::2
2603:1026:6:29::2
2603:1026:c02:4010::2
52.97.183.130
40.101.72.162
40.101.72.242
40.101.72.130
Aliases: smtp.office365.com
outlook.office365.com
outlook.ha.office365.com
outlook.ms-acdc.office.com

And that list of addresses probably changes all of the time – so do you want to manage that in your firewall(s) rules and in the code/configuration of your application? It would be better to use the abtsraction provided by the FQDN.

Network Rules allow you to do this now, but you must first enable DNS in the firewall.

Azure Firewall DNS

With this feature enabled, the Azure Firewall can support FQDNs in the Network Rules, opening up the possibility of using any of the supported protocol/port combinations, expanding your name-based rules beyond just HTTP/S and SQL.

By default, the Azure Firewall will use Azure DNS. That’s “OK” for traffic that will only ever be outbound and simple. But life is not normally that simple unless you host a relatively simple point solution behind your firewall. In reality:

  • You might want to let on-premises/remote locations connect to Private Endpoint-enabled PaaS services via site-to-site networking.
  • You might hit an interesting issue, which I will explain in a moment.

Before I move on, for the Private Endpoint scenario:

  1. Configure DNS servers (VMs) on you VNet configuration
  2. Configure conditional forwarders for each Private Endpoint DNS Zone to forward to Azure Private DNS via 168.63.129.16, the virtual public IP address that is used to facilitate a communication channel to Azure platform resources.
  3. Set the Azure Firewall DNS Server settings to point at these DNS servers
  4. Route traffic from your site-to-site gateway(s) to the firewall.

Split DNS Results

If two different machines attempt to resolve smtp.office365.com they will end up with different IP addresses – as you can see in the below diagram.

The result is that the client attempts to connect to smtp.office365.com on IP address A, and the firewall is permitting access on IP address B, so the connection attempt is denied.

DNS Proxy

To overcome this split DNS result (the Firewall and client getting two different resolved IP addresses for the FQDN) we can use DNS Proxy.

The implementation is actually pretty simple:

  1. Your firewall is set up to use your preferred DNS server(s).
  2. You enable DNS Proxy (a simple on/off setting).
  3. You configure your VNet/resources to use the Azure Firewall as their DNS server.

What happens now? Every time your resource attempts to resolve a DNS FQDN, it will send the request to the Azure Firewall. The Azure Firewall proxies/relays the request to your DNS server(s) and waits for the result which is cached. Now when your resource attempts to reach smtp.office365.com, it will use the IP address that the firewall has already cached. Simples!

And yes, this works perfectly well with Active Directory Domain Controllers running as DNS servers in a VNet or spoke VNet as long as your NSG rules allow the firewall to be a DNS client.

Azure App Service, Private Endpoint, and Application Gateway/WAF

In this post, I will share how to configure an Azure Web App (or App Service) with Private Endpoint, and securely share that HTTP/S service using the Azure Application Gateway, with the optional Web Application Firewall (WAF) feature. Whew! That’s lots of feature names!

Background

Azure Application (App) Services or Web Apps allows you to create and host a web site or web application in Azure without (directly) dealing with virtual machines. This platform service makes HTTP/S services easy. By default, App Services are shared behind a public/ & shared frontend (actually, load-balanced frontends) with public IP addresses.

Earlier this year, Microsoft released Private Link, a service that enables an Azure platform resource (or service shared using a Standard Tier Load Balancer) to be connected to a virtual network subnet. The resource is referred to as the linked resource. The linked resource connects to the subnet using a Private Endpoint. There is a Private Endpoint resource and a special NIC; it’s this NIC that shares the resource with a private IP address, obtained from the address space of the subnet. You can then connect to the linked resource using the Private Endpoint IPv4 address. Note that the Private Endpoint can connect to many different “subresources” or services (referred to as serviceGroup in ARM) that the linked resource can offer. For example, a storage account has serviceGroups such as file, blob, and web.

Notes: Private Link is generally available. Private Endpoint for App Services is still in preview. App Services Premium V2 is required for Private Endpoint.

The Application Gateway allows you to share/load balance a HTTP/S service at the application layer with external (virtual network, WAN, Internet) clients. This reverse proxy also offers an optional Web Application Firewall (WAF), at extra cost, to protect the HTTP/S service with the OWASP rule set and bot protection. With the Standard Tier of DDoS protection enabled on the Application Gateway virtual network, the WAF extends this protection to Layer-7.

Design Goal

The goal of this design is to ensure that all HTTP/S (HTTPS in this example) traffic to the Web App must:

  • Go through the WAF.
  • Reverse proxy to the App Service via the Private Endpoint private IPv4 address only.

The design will result in:

  • Layer-4 protection by an NSG associated with the WAF subnet. NSG Traffic Analytics will send the data to Log Analytics (and optionally Azure Sentinel for SIEM) for logging, classification, and reporting.
  • Layer-7 protection by the WAF. If the Standard Tier of DD0S protection is enabled, then the protection will be at Layer-4 (Application Gateway Public IP Address) and Layer-7 (WAF). Logging data will be sent to Log Analytics (and optionally Azure Sentinel for SIEM) for logging and reporting.
  • Connections directly to the web app will fail with a “HTTP Error 403 – Forbidden” error.

Note: If you want to completely prevent TCP connections to the web app then you need to consider App Service Environment/Isolated Tier or a different Azure platform/IaaS solution.

Design

Here is the design – you will want to see the original image:

There are a number of elements to the design:

Private DNS Zone

You must be able to resolve the FQDNs of your services using the per-resource type domain names. App Services use a private DNS zone called privatelink.azurewebsites.net. There are hacks to get this to work. The best solution is to create a central Azure Private DNS Zone called privatelink.azurewebsites.net.

If you have DNS servers configured on your virtual network(s), associate the Private DNS Zone with your DNS servers’ virtual network(s). Create a conditional forwarder on the DNS servers to forward all requests to privatelink.azurewebsites.net to 168.63.129.16 (https://docs.microsoft.com/en-us/azure/virtual-network/what-is-ip-address-168-63-129-16). This will result in:

  1. A network client sending a DNS resolution request to your DNS servers for *.privatelink.azurewebsites.net.
  2. The DNS servers forwarding the requests for *.privatelink.azurewebsites.net to 168.63.129.16.
  3. The Azure Private DNS Zone will receive the forwarded request and respond to the DNS servers.
  4. The DNS servers will respond to the client with the answer.

App Service

As stated before the App Service must be hosted on a Premium v2 tier App Service Plan. In my example, the app is called myapp with a default URI of https://myapp.azurewebsites.net. A virtual network access rule is added to the App Service to permit access from the subnet of the Application Gateway. Don’t forget to figure out what to do with the SCM URI for DevOps/GitHub integration.

Private Endpoint

A Private Endpoint was added to the App Service. The service/subresource/serviceGroup is sites. Automatically, Microsoft will update their DNS to modify the name resolution of myapp.azurewebsites.net to resolve to myapp.privatelink.azurewebsites.net. In the above example, the NIC for the Private Endpoint gets an IP address of 10.0.64.68 from the AppSubnet that the App Service is now connected to.

Add an A record to the Private DNS Zone for the App Service, resolving to the IPv4 address of the Private Endpoint NIC. In my case, myapp.privatelink.azurewebsites.net will resolve to 10.0.64.68. This in turn means that myapp.azurewebsites.net > myapp.privatelink.azurewebsites.net > 10.0.64.68.

Application Gateway/WAF

  1. Add a new Backend Pool with the IPv4 address of the Private Endpoint NIC, which is 10.0.64.68 in my example.
  2. Create a multisite HTTPS:443 listener for the required public URI, which will be myapp.joeelway.com in my example, adding the certificate, ideally from an Azure Key Vault. Use the public IP address (in my example) as the frontend.
  3. Set up a Custom Probe to test https://myapp.azurewebsites.net:443 (using the hostname option) with acceptable responses of 200-399.
  4. Create an HTTP Setting (the reverse proxy) to forward traffic to https://myapp.azurewebsites.net:443 (using the hostname option) using a well-known certificate (accepting the default cert of the App Service) for end-to-end encryption.
  5. Bind all of the above together with a routing rule.

Public DNS

Now you need to get traffic for https://myapp.joeelway.com to go to the (public, in my example) frontend IP address of the Application Gateway/WAF. There are lots of ways to do this, including Azure Front Door, Azure Traffic Manager, and third-party solutions. The easy way is to add an A record to your public DNS zone (joeelway.com, in my example) that resolves to the public IP address of the Application Gateway.

The Result

  1. A client browses https://myapp.joeelway.com.
  2. The client name resolution goes to public DNS which resolves myapp.joeelway.com to the public IP address of the Application Gateway.
  3. The client connects to the Application Gateway, requesting https://myapp.joeelway.com.
  4. The Listener on the Application Gateway receives the connection.
    • Any WAF functionality inspects and accepts/rejects the connection request.
  5. The Routing Rule in the Application Gateway associates the request to https://myapp.joeelway.com with the HTTP Setting and Custom Probe for https://myapp.azurewebsites.net.
  6. The Application Gateway routes the request for https://myapp.joeelway.com to https://myapp.azurewebsites.net at the IPv4 address of the Private Endpoint (documented in the Application Gateway Backend Pool).
  7. The App Service receives and accepts the request for https://myapp.azurewebsites.net and responds to the Application Gateway.
  8. The Application Gateway reverse-proxies the response to the client.

For Good Measure

If you really want to secure things:

  • Deploy the Application Gateway as WAFv2 and store SSL certs in a Key Vault with limited Access Policies
  • The NSG on the WAF subnet must be configured correctly and only permit the minimum traffic to the WAF.
  • All resources will send all logs to Log Analytics.
  • Azure Sentinel is associated with the Log Analytics workspace.
  • Azure Security Center Standard Tier is enabled on the subscription and the Log Analytics Workspace.
  • If you can justify the cost, DDoS Standard Tier is enabled on the virtual network with the public IP address(es).

And that’s just the beginning 🙂

Free Online Training – Azure Network Security

On June 19th, I will be teaching a FREE online class called Securing Azure Services & Data Through Azure Networking.

I’ve run a number of Cloud Mechanix training classes and I’ve had several requests asking if I would ever consider doing something online because I wasn’t doing the classes outside of Europe. Well … here’s your opportunity. Thanks to the kind folks at European Cloud Conference, I will be doing a 1-day training course online and for free for 20 lucky attendees.

The class, relevant to PaaS and IaaS, takes the best practices from Microsoft for securing services and data in Microsoft Azure, and teaches them based on real-world experience. I’ve been designing and implementing this stuff for enterprises and have learned a lot. The class contains stuff that people who live only in labs will not know … and sadly, based on my googling/reading, a lot of bloggers & copy/pasters fall into that bucket. I’ve learned that the basics of Azure virtual networking must be thoroughly understood before you can even attempt security. So I teach that stuff – don’t assume that you know this stuff already because I know that few really do. Then I move into the fun stuff, like firewalls, WAFs, Private Link/Private Endpoint, and more. The delivery platform will allow an interactive class – this will not be a webinar – I’ve been talking to different people to get advice on choosing the best platform for delivering this class.  I’ve some testing to do, but I think I’m set.

Here’s the class description:

Security is always number 1 or 2 in any survey on the fears of cloud computing. Networking in The Cloud is very different from traditional physical networking … but in some ways, it is quite similar. The goal of this workshop is to teach you how to secure your services and data in Microsoft Azure using techniques and designs that are advocated by Microsoft Azure. Don’t fall into the trap of thinking that networking means just virtual machines; Azure networking plays a big (and getting bigger) role in offering security and compliance with platform and data services in The Cloud.

This online class takes you all the way back to the basics of Azure networking so you really understand the “wiring” of a secure network in the cloud. Only with that understanding do you understand that small is big. The topics covered in this class will secure small/mid businesses, platform deployments that require regulatory compliance, and large enterprises:

  • The Microsoft global network
  • Availability & SLA
  • Virtual network basics
  • Virtual network adapters
  • Peering
  • Service endpoints
  • Public IP Addresses
  • VNet gateways: VPN & ExpressRoute
  • Network Security Groups
  • Application Firewall
  • Route Tables
  • Platform services & data
  • Private Link & Private Endpoint
  • Third-Party Firewalls
  • Azure Firewall
  • Monitoring
  • Troubleshooting
  • Security management
  • Micro-Segmentation
  • Architectures

Level: 400

Topic: Security

Category: IT Professionals

Those of you who have seen the 1-hour (and I rarely stuck to that time limit) conference version of this class will know what to expect. An older version of the session scored 99% at NIC 2020 in Oslo in February with a room packed to capacity. Now imagine that class where I had enough time to barely mention things and give me a full day to share my experience … that’s what we’re talking about here!

This class is one of 4 classes being promoted by the European Cloud Conference:

If you’re serious about participating, register your interest and a lucky few will be selected to join the classes.

Microsoft Ignite 2019 – Delivering Services Privately in Your VNet with Azure Private Link

Speakers:

  • Narayan Annamalai, Group Program Manager, Microsoft
  • Sumeet Mittal, Senior Program Manager, Microsoft

Private PaaS

We’ve been using Service Endpoint in addition with ACLs on the PaaS services. But this doesn’t provide an IP on the subnet. NSGs still need to allow access to all IPs of that PaaS service, e.g. all storage accounts.

Private Link maps your PaaS service into a subnet via an IP address. A private endpoint is effectively a NIC that connects to an instance of the PaaS.

Data Exfiltration Protection

Only a specific PaaS instance is mapped into your VNet subnet. So only one storage account, for example, joins your subnet via the Private Endpoint.  Trying to by pass this using clever tricks, e.g. DNS, will not work because the packets are dropped – this is data exfiltration protection.

Other clouds map an entire service, e.g. all storage accounts, to an IP address. Azure Private Endpoint maps a specific instance, e.g. a single storage account, to an IP address.

Secure Connectivity From On-Premises

Before:

  • You connect to PaaS via public DNS
  • The name resolves to the service public IP address
  • If VPN/no connection, you route over Internet. If ExpressRoute with Microsoft Peering enabled, you route over the ExpressRoute circuit.

After:

  • You connect to the PaaS service using a new DNS name
  • You route over the network connection (VPN/ER) to the VNet/subnet.
  • You connect to the Private Endpoint private IP address for the instance of the PaaS service.

Not Just For PaaS

Not just a new feature. It’s a new platform ability.

You can build your own services too, behind Standard Tier Load Balancer, and present the services to other VNets/tenants via Azure Private Link.

Private Link is the product. Private Endpoint is how you use it.

There are three kinds of Private Link:

  • PaaS
  • Other partner services (Snowflake)
  • Consume your own services

Simple Example – VM to Storage Account

VM sends a packet to Blob1.core.windows.net. The packet drops to the host SDN. An encapsulation layer adds a routable DIP (data center IP) address and some metadata. The packet travels the backbone network to the storage frontend. It is decapsulated and presented to the destination.

Demo

Creates a storage account. In networking, he has a choice of Public endpoint (all networks), public endpoint (selected networks), private endpoint.

He creates a private endpoint and selects the VNet/subnet. He then integrates with a private Azure DNS zone. It creates a DNS record for the storage account mapping to it’s private IP address in the VNet/subnet.

Into the storage account > Private Endpoint Connection. Tries to connect to the storage account from Internet – no access. He starts up a VM in the same VNet as the storage account private endpoint. He does a nslookup of the storage account’s private DNS name and it resolves to the IP address in the VNet.

In the VM he opens storage explorer and edits a blob. He logs into another VM that is also on the VNet. Browses to the storage blob in Storage Explorer. Opens the previously edited blob and can see the edits.

This storage account is now accessible from the VNet and nowhere else.

Announcements

  • Preview in all regions
    • Storage, ADLSv2, SQL DB, SQL DQW, Customer Own Service
  • Public preview Private Link available for Cosmos DB
    • Resions: uswestcentral, usnorth, uswest

Your Own Services

You can provide or consume your own services via Private Link.

  • Create/convert your existing services into Private Link Service – one API call to convert
  • VNet-Vnet connectivity without worrying about overlapping IP space
  • No regional, teant, subscription, or RBAC restrictions
  • More

Create Private Link Service

Lots of Marketplace scenarios spanning tenants.

  • App behind Std Load Balancer
  • Link service with one button/API call.
  • Mapped to the private IP of the load balancer

Consume Private Link Service

Similar to consuming PaaS:

  • Create a private endpoint
  • Attach to identification from the service provider
  • Done!

Approval Workflow

  1. Service provider creates a service
  2. Service provider creates a private link with Std LB
  3. Service provider shares private link service ID with consumers.
  4. Consumer creates a private endpoint in subnet with the service ID
  5. There is an approval by service provider.
  6. Consumer configures DNS to map to the private endpoint

Key Capabilities

  • Alias: Masks service provider resource IDs with a globally unique readable name. Mapped on the backend. The unique name uses a hash of stuff from service provider and other attributes.
  • Visibility: How to control access to the alias/service, e.g. stop random people DOSing you. You can make a service completely private to approved customers. Or you can make a service private to all Azure customers. Or you can limit visibility to selected subscriptions.
  • Auto-Approval: For huge services, you can avoid manual approvals. You can set audiences in the auto-approval list.
  • NAT-IP: The service provider masks customer IPs using NAT IP.

NAT IP

IP allocated by service provider

Acts as a source IP for inbound packets

Keeps service provider network private

Helps ensure overlapping IPs between source and destination are acceptable

TCP Proxy v2 Support

Service provider says they want to receive the TCP headers and extract the information. This allows you to identify unique customers even though they share a NAT IP on the service provider side: ACL, filter, etc.

Simplified Network Management

  • Predictable IP for configuring your policies
  • Cleaner ACLs on both Azure and on-prem
  • Resource the traffic the way you want
  • Approval workflow based modelling. No RBAC dependency
  • More

Demo

Goes to Private Link Center. Creates a new service and names it/selects region. Chooses the Std Load Balancer in front of the service. Selects up the frontend IP and NAT IP address. Chooses the auto-approval method.  A new Private Link Service resource is created – opens it and we can see the alias – copies it.

Creates a new Private Endpoint in a different subscription. Chooses the option to connect to the Alias ID and adds request text. Selects the VNet/subnet to put the private endpoint into.

In Private Endpoint Connections, the service provider sees the request and can approve/reject it – approves it.

On the consumer side he tries to connect to the private IP address – it’s just NATing RDP to the VMs in the service provider network.

Marketplace Services

  • Create the services
  • Advertise
  • Manage

Private Connections to Azure PaaS Services

In this post, I’d like to explain a few options you have to get secure/private connections to Azure’s platform-as-a-service offerings.

Express Route – Microsoft Peering

 

ExpressRoute comes in a few forms, but at a basic level, it’s a “WAN” connection to Azure virtual networks via one or more virtual network gateways; Customers this private peering to connect on-premises networks to Azure virtual networks over an SLA-protected private circuit. However, there is another form of peering that you can do over an ExpressRoute circuit called Microsoft peering. This is where you can use your private circuit to connect to Microsoft cloud services that are normally connected to over the public Internet. What you get:

  • Private access to PaaS services from your on-premises networks.
  • Access to an entire service, such as Azure SQL.
  • A wide array of Azure and non-Azure Microsoft cloud services.

FYI, Office 365 is often mentioned here. In theory, you can access Office 365 over Microsoft peering/ExpressRoute. However, the Office 365 group must first grant you permission to do this – the last I checked, you had to have legal proof of a regulatory need for private access to Cloud services. 

Service Endpoint

Imagine that you are running some resources in Azure, such as virtual machines or App Service Environment (ASE); these are virtual network integrated services. Now consider that these services might need to connect to other services such as storage accounts, Azure SQL, or others. Normally, when a VNet connected resource is communicating with, say, Azure SQL, the packets will be routed to “Internet” via the 0.0.0.0/0 default route for the subnet – “Internet” is everywhere outside the virtual network, not necessarily The Internet. The flow will hit the “public” Azure backbone and route to the Azure SQL compute cluster. There are two things about that flow:

  • It is indirect and introduces latency.
  • It traverses a shared network space.
  • A growing number of Azure-only services that support service endpoints.

A growing number of services, including storage accounts, Azure SQL, Cosmos DB, and Key Vault, all have services endpoints available to them. You can enable a service endpoint anywhere in the route from the VM (or whatever) to “Internet” and the packets will “drop” through the service endpoint to the required Azure service – make sure that any firewall in the service accepts packets from the private subnet IP address of the source (VM or whatever). Now you have a more direct and more private connection to the platform service in Azure from your VNet. What you get:

  • Private access to PaaS services from your Azure virtual networks.
  • Access to an entire service, such as Azure SQL, but you can limit this to a region.

Service Endpoint Trick #1

Did you notice in the previous section on service endpoints that I said:

You can enable a service endpoint anywhere in the route from the VM (or whatever) to “Internet”

Imagine you have a complex network and not everyone enables service endpoints the way that they should. But you manage the firewall, the public IPs, and the routing. Well, my friend, you can force traffic to support Azure platform services via service endpoints. If you have a firewall, then your routes to “Internet” should direct outbound traffic through the firewall. In the firewall (frontend) subnet, you can enable all the Azure service endpoints. Now when packets egress the firewall, they will “drop” through the service endpoints and to the desired Azure platform service, without ever reaching “Internet”.

Service Endpoint Trick #2

You might know that I like Azure Firewall. Here’s a trick that the Azure networking teams shared with me – it’s similar to the above one but is for on-premises clients trying to access Azure platform services.

You’ve got a VPN connection to a complex virtual network architecture in Azure. And at the frontend of this architecture is Azure Firewall, sitting in the AzureFirewallSubnet; in this subnet you enabled all the available service endpoints. Let’s say that someone wants to connect to Azure SQL using Power BI on their on-premises desktop. Normally that traffic will go over the Internet. What you can do is configure name resolution on your network (or PC) for the database to point at the private IP address of the Azure Firewall. Now Power BI will forward traffic to Azure Firewall, which will relay you to Azure SQL via the service endpoint. What you get:

  • Private access to PaaS services from your on-premises or Azure networks.
  • Access to individual instances of a service, such as an Azure SQL server
  • A growing number of Azure-only services that support service endpoints.

Private Link

In this post, I’m focusing on only one of the 3 current scenarios for Private Link, which is currently in unsupported preview in limited US regions only, for limited platform services – in other words, it’s early days.

This approach aims to give a similar solution to the above “Service Endpoint Trick #2” without the use of trickery. You can connect an instance of an Azure platform service to a virtual network using Private Link. That instance will now have a private IP address on the VNet subnet, making it fully routable on your virtual network. The private link gets a globally unique record in the Microsoft-managed privatelink.database.windows.net DNS zone. For example, your Azure SQL Server would now be resolvable to the private IP address of the private link as yourazuresqlsvr.privatelink.database.windows.net. Now your clients, be the in Azure or on-premises, can connect to this DNS name/IP address to connect to this Azure SQL instance. What you get:

  • Private access to PaaS services from your on-premises or Azure networks.
  • Access to individual instances of a service, such as an Azure SQL server.
  • (PREVIEW LIMITATIONS) A limited number of platform services in limited US-only regions.