App Services

Azure Infrastructure Announcements – October 2023

October is the month of ghosts and monsters – Halloween (Samhain as we Celts originally called it). Read on to find out what ghouls and creatures crawled from under the floorboards of Azure Infrastructure during October 2023.

App Service: Backup and Restore over Azure Virtual Network

Customers can now use custom backups, to back up their web app’s files and configuration data to a firewall-protected storage account if the following requirements are fulfilled:

The app is integrated with a virtual network, or the app is in a v3 App Service environment.
The storage account has granted access from the virtual network that the app is integrated with, or that the v3 App Service environment is created with.

I’ve noticed a few recent changes where services are enabling features to work with storage accounts over a virtual network. I would not be shocked to see the default connectivity settings of a storage account to prefer a virtual network in the future.

Azure Backup

Generally available: Enhanced soft delete for Azure Backup

Enhanced soft delete provides an improvement to the soft delete capability in Azure Backup that enables you to recover your backup data in case of accidental or malicious deletion. With enhanced soft delete, you get the ability to make soft delete always-on and irreversible, thus protecting it from being disabled by any malicious actors.

The Azure Backup team has put a lot of emphasis on rogue users and APTs attacking backups for many years. That work continues.

Generally Available: Multi-user authorisation for Backup vaults

MUA for Azure Backup adds an additional layer of protection for critical operations on your Backup vaults, providing greater security for your backups. To provide multi-user authorisation, Backup uses a resource guard to ensure critical operations are performed with proper authorisation, similar to how multi-user authorisation currently works for Recovery Services vaults.

It makes sense that the newer Azure Backup resource type gets some of the same protections as the older RSV.

Private Preview: Regional Disaster Recovery by Azure Backup for AKS

Azure Backup for AKS enables customers to protect their containerized workloads along with application data deployed on AKS clusters. The solution allows you to configure scheduled backups of your AKS clusters and restore them in same or alternate cluster in the scenarios like Operational Recovery, Accidental Deletion and Application Migration. Customers are also looking to utilize their AKS backups to recover application during a regional disaster recovery and also follow industry-wide best practice of 3-2-1 backup strategy.

The important bit is that data is being backed up, not just application code that can be redeployed from repos/container repositories.

Miscellaneous

General availability: Microsoft Azure now available from new cloud region in Italy

The Italy North datacenter region includes Azure Availability Zones, which offer you additional resiliency for your applications by designing the region with unique physical datacenter locations with independent power, network, and cooling for additional tolerance to datacenter failures.

This is one of the regions that does not have a paired region. Therefore this will impact resources that only replicate to paired regions: RSVs, storage accounts, and so on.

General availability: Microsoft Azure available from new cloud region in Israel

The first hyperscale datacenter region in Israel is ~~now~~ not available yet.

This post was released and withdrawn.

EDIT: I’m told that the region is available. In a story by Data Center Dynamics, they asked Microsoft to comment but they had not gotten a response by the time the story was posted.

Key Vault

Private Preview : Azure Managed HSM Backup/Restore when Storage Account is Behind a Private Endpoint

We are excited to announce the Private Preview of support for Azure Key Vault Managed HSM backup/restore when the storage account is behind a private endpoint. By becoming a Microsoft Trusted Service, we have enhanced our backup flow by allowing for private endpoint connection to Azure Storage accounts while backing up and restoring Managed HSM resources. This helps reduce the risk of exposure to public internet and helps address compliance needs.

Now you need to secure that storage account!

Private Preview: Azure Managed HSM Integration with Azure Backup

We are excited to announce the Private Preview of Azure Key Vault Managed HSM integration with Azure Backup which greatly enhances the customer experience of backing up and restoring a Managed HSM. This integration allows the process of backup and restore to take less time and effort. It is a zero-infrastructure solution with Azure Backup service managing the backups with automated retention and backup scheduling.

And how will you secure Azure Backup?

Hybrid

Introducing Azure management capabilities for Azure Arc-enabled SCVMM

Arc-enabled SCVMM allows customers to connect their SCVMM environment to Azure, enabling them to discover, onboard at-scale, and perform VM lifecycle operations from Azure on their SCVMM managed VMs.

Oh my – there really are monsters at Halloween. I forgot that SCVMM was a thing.

Azure Monitor

Migrate to Azure Monitor agent–based VM insights by 31 August 2024

On 31 August 2024, we’ll retire Log Analytics agent–based VM insights. Migrate to Azure Monitor agent–based VM insights, which offers improvements such as:

Enhanced security and performance.
Data collection rules to help cut costs.
Simplified management experience, including efficient troubleshooting.

There have been things to delay this migration but they have been cleared up now. It’s time to switch and migrate.

Azure Activity Logs Legacy solution is replaced by Diagnostic settings

If you are using automation to enable or disable the connector using the Legacy API, please note that the automation will not be supported, and you will need to recreate it using the new API.

I think this only affects people who configured the settings via API only.

Announcing General Availability: Azure Change Tracking & Inventory using Azure Monitor agent (AMA)

We are excited to announce the general availability to configure Azure Change Tracking & Inventory using the Azure Monitor agent (AMA).

Speaking of things that have been cleared up …

Virtual Machines

Public Preview Announcement: Azure VMSS Zonal Expansion

We’re thrilled to introduce the public preview of VMSS Zonal Expansion. This feature enables you to take regionally (non-zonal) configured VMs and distribute across Azure availability zones in a zonal configuration, enhancing your business continuity and resilience with minimal disruption and potentially increasing your availability SLA from 99.95% to 99.99%.

Long story short: If you have stateless VMSS workloads, then new instances will be spread across availability zones if you enable this feature. You can clean up old instances to return the VMSS to the desired scale. Service Fabric, AKS, and stateful workloads are not supported.

Breaking Change Notice: Virtual Machine Scale Set Default Orchestration Mode changing from Uniform to Flexible on PowerShell, Azure CLI

We are announcing an upcoming breaking change to the default orchestration mode for Virtual Machine Scale Sets created on PowerShell and Azure CLI starting November 2023. Once this change is complete, any new VM scale sets created using these clients will automatically default to Flexible orchestration mode instead of Uniform.

Check the defaults being used in your scripts and templates.

Generally Available: Azure Dedicated Host – Resize

You can now automatically create the new host, migrate all your existing VMs, and delete your old host, thus avoiding any manual operations when upgrading your dedicated host. This may also result in cost savings, giving you the ability to run more VMs on the new dedicated host SKUs.

Both customers are delighted.

Public Preview: VMSS Automatic Instance Repairs – Reimage, Restart Repair Actions

We are excited to announce that customers can now choose between Replace, Reimage (Preview), or Restart (Preview) as the default repair action performed in response to an “Unhealthy” application signal. These new options provide a less-impactful repair process, ensuring higher application availability while preserving VM properties and metadata for customers with sensitive workloads.

Useful I guess.

Default outbound access for VMs in Azure will be retired— updates and more information

As noted, for enhanced security, Azure is moving towards a secure-by-default model. This means default outbound access to the internet will be turned off. After 30 September 2025, Azure will no longer assign a default implicit IP for VMs to communicate to the internet. Existing VMs will not be impacted by this retirement.

See my post for more information.

The availability of Azure compute reservations exchanges extended until at least July 1st, 2024

Through a grace period, you will have the ability to exchange Azure compute reservations (Azure Reserved Virtual Machine Instances, Azure Dedicated Host reservations, and Azure App Services reservations) until at least July 1, 2024.

This change was hinted at in the docs when reserved instances were launched. It looks like Microsoft is giving customers more time to trade in to switch to Savings Plan For Compute, which can guarantee Microsoft more money.

GA: Bring our own keys (BYOK) on Ephemeral OS Disk

You can now encrypt OS and Data disks of your AKS nodes in production with your own managed keys.

Ephemeral disks are great for workloads where there is a stateless tier of VMs and you want to reset them frequently/quickly. This is an extra level of security with isolation from the provider.

Public preview: Announcing the new Azure Bastion Developer SKU

Bastion Developer allows users to establish secure connections to a single VM at a time without the necessity of additional network configurations or exposing public IPs on VMs. Users can directly access their VMs through the connect experience on the VM blade in portal, with RDP/SSH access already available and CLI-based SSH access coming soon. Bastion Developer caters to Dev/Test users seeking secure VM connections without the need for additional features or scalability.

It feels like a solution for “sandbox” scenarios rather than an actual dev/test on production VNets in a hub & spoke.

Networking

ExpressRoute Traffic Collector is now generally available

ExpressRoute Traffic Collector enables you to capture information about IP flows sent over ExpressRoute direct circuits. You can enable flow logs capture for both Private and Microsoft peering with ExpressRoute Traffic Collector. Captured flow logs data get sent to a Log Analytics workspace where you can create your own log queries for further analysis.

You can use this data for:

Monitoring/diagnostics: Use the log data to understand flows, troubleshoot, and forecast.
Cost management: figure out what share of traffic a workload is responsible for and divide the cost by that.
Security: Plug the data into your SIEM

Announcing Public Preview of Azure API Management Pricing Tiers: Basic v2 and Standard v2

We’re thrilled to announce the public preview launch of our latest Azure API Management pricing tiers: Basic v2 and Standard v2. These new tiers address highly sought-after customer requests, bring quality-of-service enhancements, and offer a flexible starting point for API Management, allowing organizations of any size to embark on their API journey.

Finn, APIM isn’t networking! The new tiers support networking options for isolating the network traffic to and from your API Management service instance. Standard v2 now supports VNet Integration, enabling outbound traffic to be restricted to a single connected VNet. As they get close to general availability, both Basic v2 and Standard v2 will support Private Link for securing inbound and outbound API traffic.

How Azure is keeping customers secure against the Rapid Reset DDoS Vulnerability

Bad actors can expose a new security vulnerability to initiate a DDoS attack on a customer’s infrastructure. This attack is leveraged against servers implementing the HTTP/2 protocol. Windows, .NET Kestrel, and HTTP .Sys (IIS) web servers are also impacted by the attack. Azure Guest Patching Service keeps customers secure by ensuring the latest security and critical updates are applied using Safe Deployment Practices on their VM and VM Scale Sets.

Make sure that your VMs or VMSSs are patching. WAF limiting rules should help but with so many required exceptions, I wonder if WAF does much!

General availability: Default Rule Set 2.1 for regional WAF with Application Gateway

DRS 2.1 is baselined off the Open Web Application Security Project (OWASP) Core Rule Set (CRS) 3.3.2 and extended to include additional proprietary protections rules developed by Microsoft Threat Intelligence team. The Microsoft Threat Intel team analyzes Common Vulnerabilities and Exposures (CVEs) and further adapts the CRS ruleset to address CVEs and reduce false positives.

Who knows when I’ll get to try this out in production – you can’t just switch OWASP rulesets because bat-sh*t bad stuff can happen to your workloads. I wonder if it resolves any issues regarding false positives which many overrides.

Code

General Availability of Azure Automation extension for Visual Studio Code

We’ve built this experience for customers who heavily use the PowerShell ISE for writing runbooks instead of using the built-in browser-based interface for runbook authoring, while working with PowerShell cmdlets and Automation Assets. With a leg up to the current browser-based interface, the extension makes runbook developers more productive and reduces the E2E time for runbook management.

This might be useful for those that regularly write or edit automation runbooks I do it about once a year 🙂

Windows Virtual Desktop

OneDrive support for Azure Virtual Desktop RemoteApps in public preview

With this new support, you can now use Microsoft OneDrive alongside a RemoteApp in Azure Virtual Desktop, allowing users to access and synchronize their files while using a RemoteApp. When a user connects to a RemoteApp, OneDrive can automatically launch as a companion to the RemoteApp. The new support has the same features and usability as the OneDrive on your personal device.

I cannot imagine using O365 without OneDrive. How could it have taken this long?

Azure Infrastructure Announcements – September 2023

September is a month of storms. There appears to have been lots of activity in the Azure cloud last month too. Everyone working on Azure should pay attention to the PAY ATTENTION! section.

PAY ATTENTION!

Default outbound access for VMs in Azure will be retired— transition to a new method of internet access

On 30 September 2025, default outbound access connectivity for virtual machines in Azure will be retired. After this date, all new VMs that require internet access will need to use explicit outbound connectivity methods such as Azure NAT Gateway, Azure Load Balancer outbound rules, or a directly attached Azure public IP address.

There will be more communications on this from Microsoft. But this is more than a “don’t worry about your existing VMs” situation. What happens when you add more VMs to an existing old network? What happens when you do a restore? What happens when you do an Azure Site Recovery failover? Those are all new VMs in old networks and they are affected. Everyone should do some work to see if they are affected and prepare remediations in advance – not on the day when they are stressed out by a restore or a Black Friday expansion.

App Service Environment version 1 and version 2 will be retired on 31 August 2024

After 31 August 2024, App Service Environment v1 and v2 will no longer be supported and these App Service Environments and the applications running on them will be deleted and any application data associated with them will be lost.

Oh yeah, you’d better start working on migrations now.

Azure Kubernetes Service

Application gateway for Containers vs Application Gateway Ingress Controller – What’s changed?

Application Gateway for Containers is a new application (layer 7) load balancing and dynamic traffic management product for workloads running in a Kubernetes cluster. At the time of writing this service is currently in public preview. In this article we will look at the differences between AGIC and Application Gateway for containers and some of the great new features available through this new offering.

I know little about AKS but this subject seems to have excited some AKS users.

A Bucket Load Of Stuff

Too much for me to get into and I don’t know enough about this stuff:

App Services

Announcing Public Preview of Free Hosting Plan for WordPress on App Service

We announced the General Availability of WordPress on App Service one year ago, in August 2022 with 3 paid hosting plans. We learnt that sometimes you might need to try out the service before you migrate your production applications. So, we are offering you a playground for a limited period – a free hosting plan to and explore and experiment with WordPress on App Service. This will help you understand the offering better before you make a long-term investment.

They really want you to try this out – note that this plan is not for production workloads.

Hybrid

Announcing the General Availability of Jumpstart HCIBox

Almost one year ago the Jumpstart team released the public preview of HCIBox, our self-contained sandbox for exploring Azure Stack HCI capabilities without the need for physical hardware. Feedback from the community has been fantastic, with dozens of feature requests and issues submitted and resolved through our open-source community.

Today, the Jumpstart team is excited to announce the general availability of HCIBox!

It’s one thing to test out the software functionality of Azure Stack HCI. But the reality is that this is a hardware-centric solution and there is no simulating the performance, stability, or operations of something this complex.

Generally Available: Windows Server 2012 and 2012 R2 Extended Security Updates enabled by Azure Arc

Windows Server 2012 and 2012 R2 Extended Security Updates (ESUs) enabled by Azure Arc is now Generally Available. Windows Server 2012 and 2012 R2 are going End of Support on October 10, 2023. With ESUs, customers who are running Windows Server 2012 on-premises or in other clouds can get three more years of critical security updates from Microsoft to protect their End of Life infrastructure.

This is not free. This is tied into the news about Azure Update Manager (below).

Miscellaneous

Detailed CSP to EA Migration guidance and crucial considerations

In this blog, I’ve shared insights drawn from real-world migration experiences. This article can help you meticulously plan your own CSP to EA migration, ensuring a smoother transition while incorporating critical considerations into your migration strategy.

One really wishes that CSP, EA, etc were just differences in billing and not Azure APIs. Changing of billing should be like changing a phone plan.

Top 10 Considerations for running your workload successfully on Azure this Holiday Season

Black Friday, Small Business Saturday and Cyber Monday will test your app’s limits, and so it’s time for your Infrastructure and Application teams to ensure that your platforms delivers when it is needed the most. Be it shopping applications on the web and mobile or payment gateways or banking systems supporting payments or inventory systems or billing systems – anything and everything associated with the shopping season should be prepared to face the load for this holiday season.

The “holiday season” starts earlier every year. Tesco Ireland started in August. Amazon has a Prime Day next Tuesday (October 10). These events test systems harder than ever and monolithic on-prem designs will not handle it. It’s time to get ready – if it’s not already too late!

Ungated Public Preview: Azure API Center

We’re thrilled to share that Azure API Center is now open for everyone to try during our ungated public preview! Azure API Center is a new Azure service that is part of the Azure API Management platform. It is the central hub where you can effortlessly keep track of all your APIs company-wide, making them readily discoverable, reusable, and manageable.

Managing a catalog of APIs could be challenging. Tooling is welcome.

Generally available: Secure critical infrastructure from accidental deletions at scale with Policy

We are thrilled to announce the general availability of DenyAction, a new effect in Azure Policy! With the introduction of Deny Action, policy enforcement now expands into blocking request based on actions to the resource. These deny action policy assignments can safeguard critical infrastructure by blocking unwarranted delete calls.

Can you believe that Azure was designed deliberately to not have a deny permission? Adding it after is not easy. The idea here is that delete locks on resources/resource groups become too easy to remove – and are frequently removed. Something, like a policy, that is enforced in the API (between you and the resources) is always applied and is not easy to remove and can be easily deployed at scale.

Virtual Machines

Generally available: Azure Premium SSD v2 Disk Storage is now available in more regions

Azure Premium SSD v2 Disk Storage is now available in Australia East, France Central, Norway East and UAE North regions. This next-generation storage solution offers advanced general-purpose block storage with the best price performance, delivering sub-millisecond disk latencies for demanding IO-intensive workloads at a low cost.

Expanded region availability makes this something more interesting. But, Azure Backup support is in very limited preview since the Spring.

Announcing the general availability of new Azure burstable virtual machines

we are announcing the general availability of the latest generations of Azure Burstable virtual machine (VM) series – the new Bsv2, Basv2, and Bpsv2 VMs based on the Intel® Xeon® Platinum 8370C, AMD EPYC™ 7763v, and Ampere® Altra® Arm-based processors respectively.

Faster and cheaper than the previous editions of B-Series VMs and they include ARM support too. The new virtual machines support all remote disk types such as Standard SSD, Standard HDD, Premium SSD and Ultra Disk storage.

Generally Available: Azure Update Manager

We are pleased to announce that Azure Update Manager, previously known as Update Management Center, is now generally available.

The controversial news is that Arc-managed machines will cost $5/month. I’m still not sold on this solution – it still feels less than legacy solutions like WSUS.

Announcing Public Preview of NVMe-enabled Ebsv5 VMs offering 400K IOPS and 10GBps throughput

Today, we are announcing a Public Preview of accelerated remote storage performance using Azure Premium SSD v2 or Ultra disk and selected sizes within the existing NVMe-enabled Ebsv5 family. The higher storage performance is offered on the E96bsv5 and E112ibsv5 VM sizes and delivers up to 400K IOPS (I/O operations per second) and 10GBps of remote disk storage throughput.

Even the largest SQL VM that I have worked with comes nowhere near these specs. The customer(s) that have justified this investment by Microsoft must be huge.

Azure savings plan for compute: How the benefit is applied

Organizations are benefiting from Azure savings plan for compute to save up to 65% on select compute services – and you could too. By committing to spending a fixed hourly amount for either one year or three years, you can save on plans tailored to your budget needs. But you may wonder how Azure applies this benefit.

It’s simple really. The system looks at your VMs, calculates the theoretical savings, and first applies your discount to the machines where you will save the most money, and then repeats until your discount is used.

General Availability: Share VM images publicly with community gallery – Azure Compute Gallery feature

With community gallery, a new feature of Azure Compute Gallery, you can now easily share your VM images with the wider Azure community. By setting up a ‘community gallery’, you can group your images and make them available to other Azure customers. As a result, any Azure customer can utilize images from the community gallery to create resources such as virtual machines (VMs) and VM scale sets.

This is a cool idea.

Trusted Launch for Azure VMware Solution virtual machines

Azure VMware Solution proudly introduces Public Preview of Trusted Launch for Virtual Machines. This advanced feature comprises Secure Boot, Virtual Trusted Platform Module (vTPM), and Virtualization-based Security (VBS), collectively forming a formidable defense against modern cyber threats.

A feature that was introduced in Windows Server 2016 Hyper-V.

Infrastructure-As-Code

Introduction to Azure DevOps Workload identity federation (OIDC) with Terraform

Workload identity federation is an OpenID Connect implementation for Azure DevOps that allow you to use short-lived credential free authentication to Azure without the need to provision self-hosted agents with managed identity. You configure a trust between your Azure DevOps organisation and an Azure service principal. Azure DevOps then provides a token that can be used to authenticate to the Azure API.

This looks like a more secure way to authenticate your pipelines. No secrets are stored and a trust between your DevOps organasation and Azure enables short-lived authentication with desired access rights/scopes.

Quickstart: Automate an existing load test with CI/CD

In this article, you learn how to automate an existing load test by creating a CI/CD pipeline in Azure Pipelines. Select your test in Azure Load Testing, and directly configure a pipeline in Azure DevOps that triggers your load test with every source code commit. Automate load tests with CI/CD to continuously validate your application performance and stability under load.

This is not something that I have played with but I suspect that you don’t want to do this against production systems!

General Availability: GitHub Advanced Security for Azure DevOps

Starting September 20th, 2023, the core scanning capabilities of GitHub Advanced Security for Azure DevOps can now be self-enabled within Azure DevOps and connect to Microsoft Defender for Cloud. Customers can automate security checks in the developer workflow using:

Code Scanning: locates vulnerabilities in source code and provides remediation guidance.
Secret Scanning: identifies high-confidence secrets and blocks developers from pushing secrets into code repositories.
Dependency Scanning: discovers vulnerabilities with open-source dependencies and automates update alerts for developers.

This seems like a good direction to go but I’m told it’s quite pricey.

Networking

General availability: Sensitive Data Protection for Application Gateway Web Application Firewall

WAF running on Application Gateway now supports sensitive data protection through log scrubbing. When a request matches the criteria of a rule, and triggers a WAF action, that event is captured within the WAF logs. WAF logs are stored as plain text for debuggability, and any matching patterns with sensitive customer data like IP address, passwords, and other personally identifiable information could potentially end up in logs as plain text. To help safeguard this sensitive data, you can now create log scrubbing rules that replace the sensitive data with “******”.

Sounds good to me!

General availability: Gateway Load Balancer IPv6 Support

Azure Gateway Load Balancer now supports IPv6 traffic, enabling you to distribute IPv6 traffic through Gateway Load Balancer before it reaches your dual-stack applications.

With this support, you can now add IPv6 frontend IP addresses and backend pools to Gateway Load Balancer. This allows you to inspect, protect, or mirror both IPv4 and IPv6 traffic flows using third-party or custom network virtual appliances (NVAs).

Useful for security architectures where NVAs are being used

Azure Backup

Preview: Cross Region Restore (CRR) for Recovery Services Agent (MARS) using Azure Backup

We are announcing the support of Cross Region Restore for Recovery Services Agent (MARS) using Azure Backup.

This makes sense. Let’s say I back up my on-prem data, located in Virginia, to Azure East US, in Boydton Virginia. And then there’s a disaster in VA that wipes out my office and Azure East US. Now I can restore to a new location from the paired region replica.

Preview: Save Azure Backup Recovery Services Agent (MARS) passphrase to Azure Key Vault

Now, you can save your Azure Recovery Services Agent encryption passphrase in Azure Key Vault directly from the console, making the Recovery Services Agent installation seamless and secure.

This beats the old default option of saving it as a text file on the machine that you were backing up.

General availability: Selective Disk Backup and Restore in Enhanced Policy for Azure VM Backup

We are adding the “Selective Disk Backup and Restore” capability in Enhanced Policy of Azure VM Backup.

Be careful out there!

Storage

General Availability: Malware Scanning in Defender for Storage

Malware Scanning in Defender for Storage will be generally available September 1, 2023.

Please make sure that you read up on how much this will cost you. The DfC plans changed recently, and the pricing model for Storage plans changed to include this feature.

Azure Monitor

Public preview: Alerts timeline view

Azure Monitor alerts is previewing a new timeline view that simplifies the consumption experience of fired alerts. The new view has the following advantages:

Shows fired alerts on a timeline
Helps identify co-occurrence of alerts
Displays alerts in the context of the resources they fired on
Focuses on showing counts of alerts to better understand impact
Supports viewing alerts by severity
Provides a more intuitive discovery and investigation path

This might be useful if you are getting a lot of alerts.

Azure Virtual Desktop

Announcing general availability of Azure Virtual Desktop Custom Image Templates

Custom image templates allow admins to build a custom “golden image” using the Azure Virtual Desktop management user interface. Leverage a variety of built-in customizations or add your own customization scripts to install applications or configurations.

Why are they not using Azure Image Builder like I do?

Cannot Remove Subnet Because of App Service VNet Integration

This post explains how to unlock a subnet when you have deleted an App Service/Function App with Regional VNet Integration.

Here I will describe how you can deal with an issue where you cannot delete a subnet from a VNet after deleting an Azure App Service or Function App with Regional VNet Integration.

Scenario

You have an Azure App Service or Function App that has Regional VNet Integration enabled to connect the PaaS resource to a subnet. You are doing some cleanup or redeployment work and want to remove the PaaS resources and the subnet. You delete the PaaS resources and then find that you cannot:

Delete the subnet
Disable subnet integration for Microsoft.Web/serverFarms

The error looks something like this:

Failed to delete resource group workload-network: Deletion of resource group ‘workload-network’ failed as resources with identifiers ‘Microsoft.Network/virtualNetworks/workload-network-vnet’ could not be deleted. The provisioning state of the resource group will be rolled back. The tracking Id is ‘iusyfiusdyfs’. Please check audit logs for more details. (Code: ResourceGroupDeletionBlocked) Subnet IntegrationSubnet is in use by /subscriptions/sdfsdf-sdfsdfsd-sdfsdfsdfsd-sdfs/resourceGroups/workload-network/providers/Microsoft.Network/virtualNetworks/workload-network-vnet/subnets/IntegrationSubnet/serviceAssociationLinks/AppServiceLink and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet. (Code: InUseSubnetCannotBeDeleted, Target: /subscriptions/sdfsdf-sdfsdfsd-sdfsdfsdfsd-sdfs/resourceGroups/workload-network/providers/Microsoft.Network/virtualNetworks/workload-network-vnet)

It turns out that deleting the PaaS resource leaves you in a situation where you cannot disable the integration. You have lost permission to access the platform mechanism.

In my situation, Regional VNet integration was not cleanly disabling so I did the next logical thing (in a non-production environment): started to delete resources, which I could quickly redeploy using IaC … but I couldn’t because the subnet was effectively locked.

Solutions

There are 2 solutions:

Call support.
Recreate the PaaS resources

Option 1 is a last resort because that’s days of pain – being frankly honest. That leaves you with Option 2. Recreate the PaaS resources exactly as they were before with Regional VNet Integration Enabled. Then disable the integration (go into the PaaS resource, go into Networking, and disconnect the integration).

That process cleans things up and now you can disable the Microsoft.Web/serverFarms delegation and/or delete the subnet.

The Azure IaaS Book Of News – December 2022

Here’s all the news that I thought was interesting for Ops and Security folks working with Azure IaaS from December 2022.

Azure VMware Solution

Azure VMware Solution Advanced Monitoring: This solution add-on deploys a virtual machine running Telegraf in Azure with a managed identity that has contributor and metrics publisher access to the Azure VMware Solution private cloud object. Telegraf then connects to vCenter Server and NSX-T Manager via API and provides responses to API metric requests from the Azure portal.

Azure Kubernetes Service

Microsoft and Isovalent partner to bring next generation eBPF dataplane for cloud-native applications in Azure: Microsoft announces the strategic partnership with Isovalent to bring Cilium’s eBPF-powered networking data plane and enhanced features for Kubernetes and cloud-native infrastructure. Azure Kubernetes Services (AKS) will now be deployed with Cilium open-source data plane and natively integrated with Azure Container Networking Interface (CNI). Microsoft and Isovalent will enable Isovalent Cilium Enterprise as a Kubernetes container App offering onto Azure Container Marketplace. This will provide a one-click deployment solution to Azure Kubernetes clusters with Isovalent Cilium Enterprise advanced features.
Generally Available: Kubernetes 1.25 support in AKS: AKS support for Kubernetes release 1.25 is now generally available. Kubernetes 1.25 delivers 40 enhancements. This release includes new changes such as the removal of PodSecurityPolicy.

Azure Backup

General Availability of Cross Zonal Restore of Azure Virtual Machines from Azure Backup: With the preview of Cross Zonal Restore of Azure VMs, Azure Backup offers a compelling set of durability options for your backup data including ZRS for intra-region high durability. Aidan’s note – you should consider this with regions such as Norway East where the paired region is unavailable to 99.9% of customers.
How to automate On-Demand Azure Backup for Azure Virtual Machines using PowerShell: Aidan’s note – A solution to enable more frequent VM backups than otherwise possible, but make sure frequency doesn’t overlap with backup job time.

Azure Virtual Desktop

Announcing the Public Preview of AVD Insights at Scale: This update provides the ability to review performance and diagnostic information across multiple host pools in one view. Aidan’s note – no additional diagnostics settings are required.
Confidential Virtual Machine support for Azure Virtual Desktop now in Public Preview: Azure Virtual Desktop has public preview support for Azure Confidential Virtual Machines. Confidential Virtual Machines increase data privacy and security by protecting data in use.
Announcing general availability of RDP Shortpath: RDP Shortpath improves the transport reliability of Azure Virtual Desktop connections by establishing a direct UDP data flow between the Remote Desktop client and session hosts. This feature is enabled by default for all customers. Aidan’s Note – I haven’t looked into this but there may be networking issues where firewall’s/routing are deployed.
Announcing general availability of FSLogix 2210: This latest version is focused on three core features, six bug fixes, and two general updates.

Virtual Machines

Public preview: New Memory Optimized VM sizes – E96bsv5 and E112ibsv5: The new E96bsv5 and E112ibsv5 VM sizes part of the Azure Ebsv5 VM series offer the highest remote storage performances of any Azure VMs to date. The new VMs can now achieve even higher VM-to-disk throughput and IOPS performance with up to 8,000 MBps and 260,000 IOPS.
Generally Available: Azure Dedicated Host – Restart: Azure Dedicated Host gives you more control over the hosts you deployed by giving you the option to restart any host. When undergoing a restart, the host and its associated VMs will restart while staying on the same underlying physical hardware.

Governance

Public preview: Use tag inheritance for cost management: You no longer need to ensure that every resource is tagged or rely on resource providers to support and emit tags in their billing pipeline for cost management. Aidan’s Note – Restricted to EA/MCA … which unreasonably sucks. The latest example of “cost management” excluding other customers.

App Services

Generally available: Static Web Apps Diagnostics: Static Web Apps diagnostics will help you diagnose what went wrong and will show you how to resolve the issues.

Storage

Public preview: Azure NetApp Files cross-zone replication: The cross-zone replication feature allows you to replicate your Azure NetApp Files volumes asynchronously from one Azure availability zone (AZ) to another in the same region.

Azure Site Recovery

Public Preview: Azure Site Recovery Higher Churn Support: Azure Site Recovery (ASR) has increased its data churn limit by approximately 2.5x to 50 MB/s per disk. With this, you can configure disaster recovery (DR) for Azure VMs having data churn up to 100 MB/s. This helps you to enable DR for more IO intensive workloads.

Networking

General availability: Feature enhancements to Azure Web Application Firewall (WAF): Azure’s global Web Application Firewall (WAF) running on Azure Front Door, and Azure’s regional WAF running on Application Gateway, now support additional features that help organizations improve their security posture and make it easier to manage logging across resources.

Miscellaneous

Public Preview : Introducing Multi-Region Replication for Azure Key Vault Managed HSM: The feature allows you to extend a managed HSM pool from one Azure region to an other thereby enhancing the availability of mission critical cryptographic keys with automated key replication and maximizing read throughput and latency with the closest available region.

Azure App Service, Private Endpoint, and Application Gateway/WAF

In this post, I will share how to configure an Azure Web App (or App Service) with Private Endpoint, and securely share that HTTP/S service using the Azure Application Gateway, with the optional Web Application Firewall (WAF) feature. Whew! That’s lots of feature names!

Background

Azure Application (App) Services or Web Apps allows you to create and host a web site or web application in Azure without (directly) dealing with virtual machines. This platform service makes HTTP/S services easy. By default, App Services are shared behind a public/ & shared frontend (actually, load-balanced frontends) with public IP addresses.

Earlier this year, Microsoft released Private Link, a service that enables an Azure platform resource (or service shared using a Standard Tier Load Balancer) to be connected to a virtual network subnet. The resource is referred to as the linked resource. The linked resource connects to the subnet using a Private Endpoint. There is a Private Endpoint resource and a special NIC; it’s this NIC that shares the resource with a private IP address, obtained from the address space of the subnet. You can then connect to the linked resource using the Private Endpoint IPv4 address. Note that the Private Endpoint can connect to many different “subresources” or services (referred to as serviceGroup in ARM) that the linked resource can offer. For example, a storage account has serviceGroups such as file, blob, and web.

Notes: Private Link is generally available. Private Endpoint for App Services is still in preview. App Services Premium V2 is required for Private Endpoint.

The Application Gateway allows you to share/load balance a HTTP/S service at the application layer with external (virtual network, WAN, Internet) clients. This reverse proxy also offers an optional Web Application Firewall (WAF), at extra cost, to protect the HTTP/S service with the OWASP rule set and bot protection. With the Standard Tier of DDoS protection enabled on the Application Gateway virtual network, the WAF extends this protection to Layer-7.

Design Goal

The goal of this design is to ensure that all HTTP/S (HTTPS in this example) traffic to the Web App must:

Go through the WAF.
Reverse proxy to the App Service via the Private Endpoint private IPv4 address only.

The design will result in:

Layer-4 protection by an NSG associated with the WAF subnet. NSG Traffic Analytics will send the data to Log Analytics (and optionally Azure Sentinel for SIEM) for logging, classification, and reporting.
Layer-7 protection by the WAF. If the Standard Tier of DD0S protection is enabled, then the protection will be at Layer-4 (Application Gateway Public IP Address) and Layer-7 (WAF). Logging data will be sent to Log Analytics (and optionally Azure Sentinel for SIEM) for logging and reporting.
Connections directly to the web app will fail with a “HTTP Error 403 – Forbidden” error.

Note: If you want to completely prevent TCP connections to the web app then you need to consider App Service Environment/Isolated Tier or a different Azure platform/IaaS solution.

Design

Here is the design – you will want to see the original image:

There are a number of elements to the design:

Private DNS Zone

You must be able to resolve the FQDNs of your services using the per-resource type domain names. App Services use a private DNS zone called privatelink.azurewebsites.net. There are hacks to get this to work. The best solution is to create a central Azure Private DNS Zone called privatelink.azurewebsites.net.

If you have DNS servers configured on your virtual network(s), associate the Private DNS Zone with your DNS servers’ virtual network(s). Create a conditional forwarder on the DNS servers to forward all requests to privatelink.azurewebsites.net to 168.63.129.16 (https://docs.microsoft.com/en-us/azure/virtual-network/what-is-ip-address-168-63-129-16). This will result in:

A network client sending a DNS resolution request to your DNS servers for *.privatelink.azurewebsites.net.
The DNS servers forwarding the requests for *.privatelink.azurewebsites.net to 168.63.129.16.
The Azure Private DNS Zone will receive the forwarded request and respond to the DNS servers.
The DNS servers will respond to the client with the answer.

App Service

As stated before the App Service must be hosted on a Premium v2 tier App Service Plan. In my example, the app is called myapp with a default URI of https://myapp.azurewebsites.net. A virtual network access rule is added to the App Service to permit access from the subnet of the Application Gateway. Don’t forget to figure out what to do with the SCM URI for DevOps/GitHub integration.

Private Endpoint

A Private Endpoint was added to the App Service. The service/subresource/serviceGroup is sites. Automatically, Microsoft will update their DNS to modify the name resolution of myapp.azurewebsites.net to resolve to myapp.privatelink.azurewebsites.net. In the above example, the NIC for the Private Endpoint gets an IP address of 10.0.64.68 from the AppSubnet that the App Service is now connected to.

Add an A record to the Private DNS Zone for the App Service, resolving to the IPv4 address of the Private Endpoint NIC. In my case, myapp.privatelink.azurewebsites.net will resolve to 10.0.64.68. This in turn means that myapp.azurewebsites.net > myapp.privatelink.azurewebsites.net > 10.0.64.68.

Application Gateway/WAF

Add a new Backend Pool with the IPv4 address of the Private Endpoint NIC, which is 10.0.64.68 in my example.
Create a multisite HTTPS:443 listener for the required public URI, which will be myapp.joeelway.com in my example, adding the certificate, ideally from an Azure Key Vault. Use the public IP address (in my example) as the frontend.
Set up a Custom Probe to test https://myapp.azurewebsites.net:443 (using the hostname option) with acceptable responses of 200-399.
Create an HTTP Setting (the reverse proxy) to forward traffic to https://myapp.azurewebsites.net:443 (using the hostname option) using a well-known certificate (accepting the default cert of the App Service) for end-to-end encryption.
Bind all of the above together with a routing rule.

Public DNS

Now you need to get traffic for https://myapp.joeelway.com to go to the (public, in my example) frontend IP address of the Application Gateway/WAF. There are lots of ways to do this, including Azure Front Door, Azure Traffic Manager, and third-party solutions. The easy way is to add an A record to your public DNS zone (joeelway.com, in my example) that resolves to the public IP address of the Application Gateway.

The Result

A client browses https://myapp.joeelway.com.
The client name resolution goes to public DNS which resolves myapp.joeelway.com to the public IP address of the Application Gateway.
The client connects to the Application Gateway, requesting https://myapp.joeelway.com.
The Listener on the Application Gateway receives the connection.
- Any WAF functionality inspects and accepts/rejects the connection request.
The Routing Rule in the Application Gateway associates the request to https://myapp.joeelway.com with the HTTP Setting and Custom Probe for https://myapp.azurewebsites.net.
- The custom probe is constantly testing the availability of https://myapp.azurewebsites.net and a request is reverse proxied if the service is available.
The Application Gateway routes the request for https://myapp.joeelway.com to https://myapp.azurewebsites.net at the IPv4 address of the Private Endpoint (documented in the Application Gateway Backend Pool).
The App Service receives and accepts the request for https://myapp.azurewebsites.net and responds to the Application Gateway.
The Application Gateway reverse-proxies the response to the client.

For Good Measure

If you really want to secure things:

Deploy the Application Gateway as WAFv2 and store SSL certs in a Key Vault with limited Access Policies
The NSG on the WAF subnet must be configured correctly and only permit the minimum traffic to the WAF.
All resources will send all logs to Log Analytics.
Azure Sentinel is associated with the Log Analytics workspace.
Azure Security Center Standard Tier is enabled on the subscription and the Log Analytics Workspace.
If you can justify the cost, DDoS Standard Tier is enabled on the virtual network with the public IP address(es).

And that’s just the beginning 🙂

Failed to add new rule: IpSecurityRestriction.VnetSubnetResourceId is invalid.

This post is focused on a scenario where you are creating an Access Restriction rule in an Azure App Service to allow client requests from a subnet in a Virtual Network (VNET) and you get this error:

Failed to add new rule: IpSecurityRestriction.VnetSubnetResourceId is invalid. For request GET https://management.azure.com/subscriptions/xxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Network/virtualNetworks/xxxxxx/taggedTrafficConsumers?api-version=2018-01-01 with clientRequestId xxxxxx and correlationRequestId xxxxxx, received a response with status code Forbidden, error code AuthorizationFailed, and response content: {“error”:{“code”:”AuthorizationFailed”,”message”:”The client ‘xxxxxx’ with object id ‘xxxxxx’ does not have authorization to perform action ‘Microsoft.Network/virtualNetworks/taggedTrafficConsumers/read’ over scope ‘/subscriptions/xxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Network/virtualNetworks/xxxxxx’ or the scope is invalid. If access was recently granted, please refresh your credentials.”}}.

The Scenario

The customer wanted to deploy Standard Tier Azure App Services with some level of security in a hub and spoke architecture. The hub is in Subscription A. There a virtual network with an Azure Application Gateway (WAG)/Web Application Firewall(WAF) is deployed into a VNET/subnet. The WAF subnet has the Microsoft.Web Service Endpoint enabled, allowing the WAF to reverse proxy web requests via the direct path of the Service Endpoint to the App Service(s).

The App Service Plan and App Services are in Subscription B. The goal is to only allow traffic to the App Services via the WAF. All the necessary DNS/SSL stuff was done and the WAF was configured to route traffic. Now, the customer wanted to prevent requests from coming in directly to the App Service – an Access Restriction rule would be created with the Virtual Network type. However, when we tried to create that rule, it failed with the above security error.

Troubleshooting

At first, we thought there was an error with Azure Privileged Identity Management (PIM), but we soon ruled that out. The customer had Contributor rights and I had Owner rights over both subscriptions and we verified access. While doing a Teams screen share the customer read an article about Azure Key Vault with a similar error that indicated an issue with Resource Providers. We both had the same idea at the same time.

Solution

In the WAF subscription, enable the Microsoft.Web resource provider. This will allow the App Service to “configure” the integration with the subnet from its own subscription and solves the security issue.

AidanFinn.Com Migrated To Azure App Services

I’ve just migrated AidanFinn.com from a Windows Server 2012 R2 Azure virtual machine to an App Service (web app) running on the same App Service Plan as CloudMechanix.com.

This site, AidanFinn.com has been running on an Azure VM for the last few years. That has given me a lot of experience with running a production workload in Azure. Azure worked well. What really irked me was MySQL, running in the VM by the way. MySQL blew up once, and wouldn’t restore, so I had to restore the entire VM. And MySQL continues to be a pain, causing the site to crash, requiring full VM reboots.

I was facing an eventual upgrade of the VM (a migration in Azure) so I made the decision to reduce my maintenance workload. I decided to switch to PaaS, and let Microsoft do the work. I previously blogged how I deployed the Cloud Mexchanix (my Azure training business) onto an Azure app service plan. I also created a stub WordPress site for AidanFinn.com, running on the same plan. The two WordPress site runs on different app services (application pools) on the same WS2016 machine, managed by Microsoft. I have auto-scaling enabled so a single (only in this case) load balanced VM instance can automatically be spun up if the CPU/memory load requires it. Both sites are using Azure Database for MySQL instances, where Microsoft looks after MySQL for me. In other words, the VM, the guest OS, and the database system are managed by Microsoft. I manage the web content. Perfect!

The migration of AidanFinn.com has always been a challenge, from it’s origins as a “Joe Elway” blog on Live Spaces all the way through to it’s previous existence as an Azure VM. I remember the bad old days of exporting and editing multiple XML files to get a migration to work once. And this time was no different. The built-in WordPress Import refused to work. I tried another third-party plug-in and that wouldn’t work. Then I tried the All-In-One WP Migration plugin. It took hours to do a 1 GB export of the content and database from the VM. When I tried to do an import, I exceeded the 512 MB free limit, so I had to pay for the professional edition ($69 or so). The import also took ages, but the site was lifted and shifted exactly as it was.

The it was time to add the custom domain names to the app service in the Azure Portal. A quick query with my DNS registrar (Blacknight) told me how to create @ records in their control panel, and I was done! I will look at hosting the domain in Azure, like I did with Cloud Mechanix, but all the Office 365 records will take time to create first.

What about the old machine? It can take up to 24 hours for DNS changes to be replicated around the world, so it will remain running until tomorrow afternoon. I have configured Auto Shutdown in the settings of the VM, with a notification to be sent to me by email first.

And that will be that! Both of my websites will be running on Azure App Services.

The App Service size is S1, costing ~€61.57/month. Each database will cost under €30 per month. Some blob storage (€0.02 per GB) is being used to backup the sites – restores have been tested! While the total is well above a $10 web hosting plan, I cannot use such plans, because I was kicked off of that platform because AidanFinn.com generated too much load. So it’s either VM or PaaS, and PaaS suits me more because there is less for me to maintain now that I am there.

Deploying My Sites On Azure App Services

I’ve started redeploying my websites on Azure App Services. In this post, I’ll explain the rather simple architecture and how I am going through my own little digital transformation or cloud transformation.

For the last few years, this site (https://aidanfinn.com) has been hosted on an Azure virtual machine running Windows Server 2012 R2 and an aging copy of MySQL. Once upon a time, before having a family, that was fine. I had lots of time, and a willingness to “muck in”. These days, I prioritise my time and my time is limited. I want to focus on content, not on admin … and isn’t that the point of the cloud?

That’s why I made the decision to switch from an IaaS virtual machine in Azure to Azure PaaS in the form of App Services, a part of Azure that has consumed a good bit of 2018 for me so far. This decision included this site, and I decided to build a new WordPress site for my Azure training business, Cloud Mechanix, on http://www.cloudmechanix.com.

The architecture is pretty simple:

Azure Database for MySQL Server

Both of my sites run on WordPress and that means MySQL – something that I know nothing about and have had problems with in the past – resulting in a complete VM restore back in the preview days of Azure Backup for IaaS VMs. If you want to know nothing about installing/running/backing up a database, then Azure Database services are for you! Many IT pros will have heard of Azure SQL, but there are also MySQL and PostgreSQL implementations of the service.

I deployed 1 instance of MySQL Server for each website. I tried to deploy 1 instance only, with multiple databases, but the second WordPress site just wasn’t having it. I’ve used the Basic tier and so far, the size seems to be OK.

A storage account was also created, and I configured diagnostics exports of both database instances to blob storage using Azure Monitor.

App Service Plan

A single app service plan hosts both websites. I decided to go with the Standard tier because I wanted backup functionality, not just custom domains that the Basic tier would offer. I offset this by being a little clever with the sizing. The plan is using a single instance (Windows Server 2016 IIS virtual machine under the covers), with content stored on a back-end SMB 3.0 share (also under the covers). I deployed the small S1 instance, keeping the costs down. However, aidanfinn.com is running on a decent spec D2s_v3 VM with 2 cores and 4 GB RAM. To offset the drop in resources and to enable peak demand, I’ve enabled autoscaling, supporting 1-2 instances. The autoscaling is configured to go to 2 instances if CPU or RAM exceeds certain thresholds for 10 minutes, and to drop to 1 instance if CPU or RAM drops back down below those thresholds.

The Cloud Mechanix site is running on App Services now, and AidanFinn.com will follow soon.

App Services

There is one app service for each website. I have deployed a storage account for backing up the websites every morning (at different times), including their databases – you can’t have enough backups!

Making The Deployment Easy

it sounds like I deployed a lot of stuff, right? Actually the WordPress, published by WordPress, template in the Azure Marketplace for App Services & Azure Database for MySQL Server made it really easy! It created the app service plan (first deployment only), the app service, and the database instance/database. Then I ran the template again, using the existing app service plan, and creating a second piring of app service and database instance/database. Then I logged into the default WordPress page of each site, and configured my credentials.

Changing the WordPress Settings URL

The default URL of the WordPress site in Settings will be configured and greyed out to use the default Azure domain name, even after you associate your own domain name with the site. The trick to changing the URL is to:

Configure the FTP password for the app service
Get the FTP username and server name from the app service properties
Connect to the app service using FTP
Browse to the /site/wwwroot folder
Download wp-config.php and edit it

Look for a line with define(‘WP_DEBUG’, false);

Straight after that, add the following line:

define(‘RELOCATE’,true);

Now:

Upload the wp-config.php file back to the site.
Browse to the new URL of the site, e.g. http://www.mynewdomain.com/wp-login.php, and sign in. This will update the WordPress URL settings of the site.
Re-download the wp-config.php file, remove the above line that you added, and upload the file again.

You’re done!