Tag: App Services

Cannot Remove Subnet Because of App Service VNet Integration

This post explains how to unlock a subnet when you have deleted an App Service/Function App with Regional VNet Integration.

Here I will describe how you can deal with an issue where you cannot delete a subnet from a VNet after deleting an Azure App Service or Function App with Regional VNet Integration.

Scenario

You have an Azure App Service or Function App that has Regional VNet Integration enabled to connect the PaaS resource to a subnet. You are doing some cleanup or redeployment work and want to remove the PaaS resources and the subnet. You delete the PaaS resources and then find that you cannot:

  • Delete the subnet
  • Disable subnet integration for Microsoft.Web/serverFarms

The error looks something like this:

Failed to delete resource group workload-network: Deletion of resource group ‘workload-network’ failed as resources with identifiers ‘Microsoft.Network/virtualNetworks/workload-network-vnet’ could not be deleted. The provisioning state of the resource group will be rolled back. The tracking Id is ‘iusyfiusdyfs’. Please check audit logs for more details. (Code: ResourceGroupDeletionBlocked) Subnet IntegrationSubnet is in use by /subscriptions/sdfsdf-sdfsdfsd-sdfsdfsdfsd-sdfs/resourceGroups/workload-network/providers/Microsoft.Network/virtualNetworks/workload-network-vnet/subnets/IntegrationSubnet/serviceAssociationLinks/AppServiceLink and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet. (Code: InUseSubnetCannotBeDeleted, Target: /subscriptions/sdfsdf-sdfsdfsd-sdfsdfsdfsd-sdfs/resourceGroups/workload-network/providers/Microsoft.Network/virtualNetworks/workload-network-vnet)

It turns out that deleting the PaaS resource leaves you in a situation where you cannot disable the integration. You have lost permission to access the platform mechanism.

In my situation, Regional VNet integration was not cleanly disabling so I did the next logical thing (in a non-production environment): started to delete resources, which I could quickly redeploy using IaC … but I couldn’t because the subnet was effectively locked.

Solutions

There are 2 solutions:

  1. Call support.
  2. Recreate the PaaS resources

Option 1 is a last resort because that’s days of pain – being frankly honest. That leaves you with Option 2. Recreate the PaaS resources exactly as they were before with Regional VNet Integration Enabled. Then disable the integration (go into the PaaS resource, go into Networking, and disconnect the integration).

That process cleans things up and now you can disable the Microsoft.Web/serverFarms delegation and/or delete the subnet.

The Azure IaaS Book Of News – December 2022

Here’s all the news that I thought was interesting for Ops and Security folks working with Azure IaaS from December 2022.

Azure VMware Solution

  • Azure VMware Solution Advanced Monitoring: This solution add-on deploys a virtual machine running Telegraf in Azure with a managed identity that has contributor and metrics publisher access to the Azure VMware Solution private cloud object. Telegraf then connects to vCenter Server and NSX-T Manager via API and provides responses to API metric requests from the Azure portal.

Azure Kubernetes Service

  • Microsoft and Isovalent partner to bring next generation eBPF dataplane for cloud-native applications in Azure: Microsoft announces the strategic partnership with Isovalent to bring Cilium’s eBPF-powered networking data plane and enhanced features for Kubernetes and cloud-native infrastructure. Azure Kubernetes Services (AKS) will now be deployed with Cilium open-source data plane and natively integrated with Azure Container Networking Interface (CNI). Microsoft and Isovalent will enable Isovalent Cilium Enterprise as a Kubernetes container App offering onto Azure Container Marketplace. This will provide a one-click deployment solution to Azure Kubernetes clusters with Isovalent Cilium Enterprise advanced features.
  • Generally Available: Kubernetes 1.25 support in AKS: AKS support for Kubernetes release 1.25 is now generally available. Kubernetes 1.25 delivers 40 enhancements. This release includes new changes such as the removal of PodSecurityPolicy.

Azure Backup

Azure Virtual Desktop

Virtual Machines

  • Public preview: New Memory Optimized VM sizes – E96bsv5 and E112ibsv5: The new E96bsv5 and E112ibsv5 VM sizes part of the Azure Ebsv5 VM series offer the highest remote storage performances of any Azure VMs to date.  The new VMs can now achieve even higher VM-to-disk throughput and IOPS performance with up to 8,000 MBps and 260,000 IOPS.
  • Generally Available: Azure Dedicated Host – Restart: Azure Dedicated Host gives you more control over the hosts you deployed by giving you the option to restart any host. When undergoing a restart, the host and its associated VMs will restart while staying on the same underlying physical hardware.

Governance

  • Public preview: Use tag inheritance for cost management: You no longer need to ensure that every resource is tagged or rely on resource providers to support and emit tags in their billing pipeline for cost management. Aidan’s Note – Restricted to EA/MCA … which unreasonably sucks. The latest example of “cost management” excluding other customers.

App Services

Storage

Azure Site Recovery

  • Public Preview: Azure Site Recovery Higher Churn Support: Azure Site Recovery (ASR) has increased its data churn limit by approximately 2.5x to 50 MB/s per disk. With this, you can configure disaster recovery (DR) for Azure VMs having data churn up to 100 MB/s. This helps you to enable DR for more IO intensive workloads.

Networking

Miscellaneous

Azure App Service, Private Endpoint, and Application Gateway/WAF

In this post, I will share how to configure an Azure Web App (or App Service) with Private Endpoint, and securely share that HTTP/S service using the Azure Application Gateway, with the optional Web Application Firewall (WAF) feature. Whew! That’s lots of feature names!

Background

Azure Application (App) Services or Web Apps allows you to create and host a web site or web application in Azure without (directly) dealing with virtual machines. This platform service makes HTTP/S services easy. By default, App Services are shared behind a public/ & shared frontend (actually, load-balanced frontends) with public IP addresses.

Earlier this year, Microsoft released Private Link, a service that enables an Azure platform resource (or service shared using a Standard Tier Load Balancer) to be connected to a virtual network subnet. The resource is referred to as the linked resource. The linked resource connects to the subnet using a Private Endpoint. There is a Private Endpoint resource and a special NIC; it’s this NIC that shares the resource with a private IP address, obtained from the address space of the subnet. You can then connect to the linked resource using the Private Endpoint IPv4 address. Note that the Private Endpoint can connect to many different “subresources” or services (referred to as serviceGroup in ARM) that the linked resource can offer. For example, a storage account has serviceGroups such as file, blob, and web.

Notes: Private Link is generally available. Private Endpoint for App Services is still in preview. App Services Premium V2 is required for Private Endpoint.

The Application Gateway allows you to share/load balance a HTTP/S service at the application layer with external (virtual network, WAN, Internet) clients. This reverse proxy also offers an optional Web Application Firewall (WAF), at extra cost, to protect the HTTP/S service with the OWASP rule set and bot protection. With the Standard Tier of DDoS protection enabled on the Application Gateway virtual network, the WAF extends this protection to Layer-7.

Design Goal

The goal of this design is to ensure that all HTTP/S (HTTPS in this example) traffic to the Web App must:

  • Go through the WAF.
  • Reverse proxy to the App Service via the Private Endpoint private IPv4 address only.

The design will result in:

  • Layer-4 protection by an NSG associated with the WAF subnet. NSG Traffic Analytics will send the data to Log Analytics (and optionally Azure Sentinel for SIEM) for logging, classification, and reporting.
  • Layer-7 protection by the WAF. If the Standard Tier of DD0S protection is enabled, then the protection will be at Layer-4 (Application Gateway Public IP Address) and Layer-7 (WAF). Logging data will be sent to Log Analytics (and optionally Azure Sentinel for SIEM) for logging and reporting.
  • Connections directly to the web app will fail with a “HTTP Error 403 – Forbidden” error.

Note: If you want to completely prevent TCP connections to the web app then you need to consider App Service Environment/Isolated Tier or a different Azure platform/IaaS solution.

Design

Here is the design – you will want to see the original image:

There are a number of elements to the design:

Private DNS Zone

You must be able to resolve the FQDNs of your services using the per-resource type domain names. App Services use a private DNS zone called privatelink.azurewebsites.net. There are hacks to get this to work. The best solution is to create a central Azure Private DNS Zone called privatelink.azurewebsites.net.

If you have DNS servers configured on your virtual network(s), associate the Private DNS Zone with your DNS servers’ virtual network(s). Create a conditional forwarder on the DNS servers to forward all requests to privatelink.azurewebsites.net to 168.63.129.16 (https://docs.microsoft.com/en-us/azure/virtual-network/what-is-ip-address-168-63-129-16). This will result in:

  1. A network client sending a DNS resolution request to your DNS servers for *.privatelink.azurewebsites.net.
  2. The DNS servers forwarding the requests for *.privatelink.azurewebsites.net to 168.63.129.16.
  3. The Azure Private DNS Zone will receive the forwarded request and respond to the DNS servers.
  4. The DNS servers will respond to the client with the answer.

App Service

As stated before the App Service must be hosted on a Premium v2 tier App Service Plan. In my example, the app is called myapp with a default URI of https://myapp.azurewebsites.net. A virtual network access rule is added to the App Service to permit access from the subnet of the Application Gateway. Don’t forget to figure out what to do with the SCM URI for DevOps/GitHub integration.

Private Endpoint

A Private Endpoint was added to the App Service. The service/subresource/serviceGroup is sites. Automatically, Microsoft will update their DNS to modify the name resolution of myapp.azurewebsites.net to resolve to myapp.privatelink.azurewebsites.net. In the above example, the NIC for the Private Endpoint gets an IP address of 10.0.64.68 from the AppSubnet that the App Service is now connected to.

Add an A record to the Private DNS Zone for the App Service, resolving to the IPv4 address of the Private Endpoint NIC. In my case, myapp.privatelink.azurewebsites.net will resolve to 10.0.64.68. This in turn means that myapp.azurewebsites.net > myapp.privatelink.azurewebsites.net > 10.0.64.68.

Application Gateway/WAF

  1. Add a new Backend Pool with the IPv4 address of the Private Endpoint NIC, which is 10.0.64.68 in my example.
  2. Create a multisite HTTPS:443 listener for the required public URI, which will be myapp.joeelway.com in my example, adding the certificate, ideally from an Azure Key Vault. Use the public IP address (in my example) as the frontend.
  3. Set up a Custom Probe to test https://myapp.azurewebsites.net:443 (using the hostname option) with acceptable responses of 200-399.
  4. Create an HTTP Setting (the reverse proxy) to forward traffic to https://myapp.azurewebsites.net:443 (using the hostname option) using a well-known certificate (accepting the default cert of the App Service) for end-to-end encryption.
  5. Bind all of the above together with a routing rule.

Public DNS

Now you need to get traffic for https://myapp.joeelway.com to go to the (public, in my example) frontend IP address of the Application Gateway/WAF. There are lots of ways to do this, including Azure Front Door, Azure Traffic Manager, and third-party solutions. The easy way is to add an A record to your public DNS zone (joeelway.com, in my example) that resolves to the public IP address of the Application Gateway.

The Result

  1. A client browses https://myapp.joeelway.com.
  2. The client name resolution goes to public DNS which resolves myapp.joeelway.com to the public IP address of the Application Gateway.
  3. The client connects to the Application Gateway, requesting https://myapp.joeelway.com.
  4. The Listener on the Application Gateway receives the connection.
    • Any WAF functionality inspects and accepts/rejects the connection request.
  5. The Routing Rule in the Application Gateway associates the request to https://myapp.joeelway.com with the HTTP Setting and Custom Probe for https://myapp.azurewebsites.net.
  6. The Application Gateway routes the request for https://myapp.joeelway.com to https://myapp.azurewebsites.net at the IPv4 address of the Private Endpoint (documented in the Application Gateway Backend Pool).
  7. The App Service receives and accepts the request for https://myapp.azurewebsites.net and responds to the Application Gateway.
  8. The Application Gateway reverse-proxies the response to the client.

For Good Measure

If you really want to secure things:

  • Deploy the Application Gateway as WAFv2 and store SSL certs in a Key Vault with limited Access Policies
  • The NSG on the WAF subnet must be configured correctly and only permit the minimum traffic to the WAF.
  • All resources will send all logs to Log Analytics.
  • Azure Sentinel is associated with the Log Analytics workspace.
  • Azure Security Center Standard Tier is enabled on the subscription and the Log Analytics Workspace.
  • If you can justify the cost, DDoS Standard Tier is enabled on the virtual network with the public IP address(es).

And that’s just the beginning 🙂

Failed to add new rule: IpSecurityRestriction.VnetSubnetResourceId is invalid.

This post is focused on a scenario where you are creating an Access Restriction rule in an Azure App Service to allow client requests from a subnet in a Virtual Network (VNET) and you get this error:

Failed to add new rule: IpSecurityRestriction.VnetSubnetResourceId is invalid. For request GET https://management.azure.com/subscriptions/xxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Network/virtualNetworks/xxxxxx/taggedTrafficConsumers?api-version=2018-01-01 with clientRequestId xxxxxx and correlationRequestId xxxxxx, received a response with status code Forbidden, error code AuthorizationFailed, and response content: {“error”:{“code”:”AuthorizationFailed”,”message”:”The client ‘xxxxxx’ with object id ‘xxxxxx’ does not have authorization to perform action ‘Microsoft.Network/virtualNetworks/taggedTrafficConsumers/read’ over scope ‘/subscriptions/xxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Network/virtualNetworks/xxxxxx’ or the scope is invalid. If access was recently granted, please refresh your credentials.”}}.

The Scenario

The customer wanted to deploy Standard Tier Azure App Services with some level of security in a hub and spoke architecture. The hub is in Subscription A. There a virtual network with an Azure Application Gateway (WAG)/Web Application Firewall(WAF) is deployed into a VNET/subnet. The WAF subnet has the Microsoft.Web Service Endpoint enabled, allowing the WAF to reverse proxy web requests via the direct path of the Service Endpoint to the App Service(s).

The App Service Plan and App Services are in Subscription B. The goal is to only allow traffic to the App Services via the WAF. All the necessary DNS/SSL stuff was done and the WAF was configured to route traffic. Now, the customer wanted to prevent requests from coming in directly to the App Service – an Access Restriction rule would be created with the Virtual Network type. However, when we tried to create that rule, it failed with the above security error.

Troubleshooting

At first, we thought there was an error with Azure Privileged Identity Management (PIM), but we soon ruled that out. The customer had Contributor rights and I had Owner rights over both subscriptions and we verified access. While doing a Teams screen share the customer read an article about Azure Key Vault with a similar error that indicated an issue with Resource Providers. We both had the same idea at the same time.

Solution

In the WAF subscription, enable the Microsoft.Web resource provider. This will allow the App Service to “configure” the integration with the subnet from its own subscription and solves the security issue.

AidanFinn.Com Migrated To Azure App Services

I’ve just migrated AidanFinn.com from a Windows Server 2012 R2 Azure virtual machine to an App Service (web app) running on the same App Service Plan as CloudMechanix.com.

Drawing1

This site, AidanFinn.com has been running on an Azure VM for the last few years. That has given me a lot of experience with running a production workload in Azure. Azure worked well. What really irked me was MySQL, running in the VM by the way. MySQL blew up once, and wouldn’t restore, so I had to restore the entire VM. And MySQL continues to be a pain, causing the site to crash, requiring full VM reboots.

I was facing an eventual upgrade of the VM (a migration in Azure) so I made the decision to reduce my maintenance workload. I decided to switch to PaaS, and let Microsoft do the work. I previously blogged how I deployed the Cloud Mexchanix (my Azure training business) onto an Azure app service plan. I also created a stub WordPress site for AidanFinn.com, running on the same plan. The two WordPress site runs on different app services (application pools) on the same WS2016 machine, managed by Microsoft. I have auto-scaling enabled so a single (only in this case) load balanced VM instance can automatically be spun up if the CPU/memory load requires it. Both sites are using Azure Database for MySQL instances, where Microsoft looks after MySQL for me. In other words, the VM, the guest OS, and the database system are managed by Microsoft. I manage the web content. Perfect!

The migration of AidanFinn.com has always been a challenge, from it’s origins as a “Joe Elway” blog on Live Spaces all the way through to it’s previous existence as an Azure VM. I remember the bad old days of exporting and editing multiple XML files to get a migration to work once. And this time was no different. The built-in WordPress Import refused to work. I tried another third-party plug-in and that wouldn’t work. Then I tried the All-In-One WP Migration plugin. It took hours to do a 1 GB export of the content and database from the VM. When I tried to do an import, I exceeded the 512 MB free limit, so I had to pay for the professional edition ($69 or so). The import also took ages, but the site was lifted and shifted exactly as it was.

The it was time to add the custom domain names to the app service in the Azure Portal. A quick query with my DNS registrar (Blacknight) told me how to create @ records in their control panel, and I was done! I will look at hosting the domain in Azure, like I did with Cloud Mechanix, but all the Office 365 records will take time to create first.

image

What about the old machine? It can take up to 24 hours for DNS changes to be replicated around the world, so it will remain running until tomorrow afternoon. I have configured Auto Shutdown in the settings of the VM, with a notification to be sent to me by email first.

And that will be that! Both of my websites will be running on Azure App Services.

The App Service size is S1, costing ~€61.57/month. Each database will cost under €30 per month. Some blob storage (€0.02 per GB) is being used to backup the sites  – restores have been tested! While the total is well above a $10 web hosting plan, I cannot use such plans, because I was kicked off of that platform because AidanFinn.com generated too much load. So it’s either VM or PaaS, and PaaS suits me more because there is less for me to maintain now that I am there.

Deploying My Sites On Azure App Services

I’ve started redeploying my websites on Azure App Services. In this post, I’ll explain the rather simple architecture and how I am going through my own little digital transformation or cloud transformation.

For the last few years, this site (https://aidanfinn.com) has been hosted on an Azure virtual machine running Windows Server 2012 R2 and an aging copy of MySQL. Once upon a time, before having a family, that was fine. I had lots of time, and a willingness to “muck in”. These days, I prioritise my time and my time is limited. I want to focus on content, not on admin … and isn’t that the point of the cloud?

That’s why I made the decision to switch from an IaaS virtual machine in Azure to Azure PaaS in the form of App Services, a part of Azure that has consumed a good bit of 2018 for me so far. This decision included this site, and I decided to build a new WordPress site for my Azure training business, Cloud Mechanix, on http://www.cloudmechanix.com.

The architecture is pretty simple:

image

Azure Database for MySQL Server

Both of my sites run on WordPress and that means MySQL – something that I know nothing about and have had problems with in the past – resulting in a complete VM restore back in the preview days of Azure Backup for IaaS VMs. If you want to know nothing about installing/running/backing up a database, then Azure Database services are for you! Many IT pros will have heard of Azure SQL, but there are also MySQL and PostgreSQL implementations of the service.

I deployed 1 instance of MySQL Server for each website. I tried to deploy 1 instance only, with multiple databases, but the second WordPress site just wasn’t having it. I’ve used the Basic tier and so far, the size seems to be OK.

A storage account was also created, and I configured diagnostics exports of both database instances to blob storage using Azure Monitor.

App Service Plan

A single app service plan hosts both websites. I decided to go with the Standard tier because I wanted backup functionality, not just custom domains that the Basic tier would offer. I offset this by being a little clever with the sizing. The plan is using a single instance (Windows Server 2016 IIS virtual machine under the covers), with content stored on a back-end SMB 3.0 share (also under the covers). I deployed the small S1 instance, keeping the costs down. However, aidanfinn.com is running on a decent spec D2s_v3 VM with 2 cores and 4 GB RAM. To offset the drop in resources and to enable peak demand, I’ve enabled autoscaling, supporting 1-2 instances. The autoscaling is configured to go to 2 instances if CPU or RAM exceeds certain thresholds for 10 minutes, and to drop to 1 instance if CPU or RAM drops back down below those thresholds.

The Cloud Mechanix site is running on App Services now, and AidanFinn.com will follow soon.

App Services

There is one app service for each website. I have deployed a storage account for backing up the websites every morning (at different times), including their databases – you can’t have enough backups!

Making The Deployment Easy

it sounds like I deployed a lot of stuff, right? Actually the WordPress, published by WordPress, template in the Azure Marketplace for App Services & Azure Database for MySQL Server made it really easy! It created the app service plan (first deployment only), the app service, and the database instance/database. Then I ran the template again, using the existing app service plan, and creating a second piring of app service and database instance/database. Then I logged into the default WordPress page of each site, and configured my credentials.

Changing the WordPress Settings URL

The default URL of the WordPress site in Settings will be configured and greyed out to use the default Azure domain name, even after you associate your own domain name with the site. The trick to changing the URL is to:

  1. Configure the FTP password for the app service
  2. Get the FTP username and server name from the app service properties
  3. Connect to the app service using FTP
  4. Browse to the /site/wwwroot folder
  5. Download wp-config.php and edit it

Look for a line with define(‘WP_DEBUG’, false);

Straight after that, add the following line:

define(‘RELOCATE’,true);

image

Now:

  1. Upload the wp-config.php file back to the site.
  2. Browse to the new URL of the site, e.g. http://www.mynewdomain.com/wp-login.php, and sign in. This will update the WordPress URL settings of the site.
  3. Re-download the wp-config.php file, remove the above line that you added, and upload the file again.

You’re done!