In this post, I’ll explain how to check if a client service, such as an App Service, has access to an Azure Key Vault with Private Endpoint.
Private Endpoint
In case you do not know, Private Endpoint gives us a mechanism where we can attach a PaaS service, such as a Key Vault, to a subnet with a NIC and a private IP address. Public connections to the PaaS resources are disabled, and an (Azure) Private DNS Zone is used to alter the name resolution of the PaaS resource to point to the private IP address.
Note that communications to the private endpoint are inbound (and response only). The PaaS resource cannot make outbound connections over a Private Endpoint.
My Scenario
The customer has an App Service Plan that has VNet Integration enabled – this allows the App Services to make outbound connections from “random” IPs on this subnet – NSG/Firewall rules should permit access from the subnet prefix.
The App Services on the plan have Private Endpoints on a second subnet in the VNet. There is also a Key Vault, which also has a Private Endpoint. The “Private Endpoint subnet” has an NSG to deny everything except desired traffic, including allowing HTTPS from the VNet Integration subnet prefix to the Key Vault Private Endpoint.
A developer was wondering if connections from an App Service were working and asked if we could see this in the logs.
Problem
The dev in this case wanted to verify network connectivity. So the obvious place to check was … the network! The way to do that is usually to verify that packets arrived at the destination NIC. You can do that (normally) using NSG Flow Logs. There is sometimes up to 25 minutes (or longer during pandemic compute shortages) of a wait before a flow appears in Log Analytics (data export from the host, 10 minutes collection interval [in our case], data processing [15 minutes]). We checked the logs but nothing was there.
And that is because (at this time) NSG Flow Logs cannot produce data destined to Private Endpoints.
We need a different way to trace connections.
Solution
The solution is to check the logs of the target resource. We enable a lot of logging by standard, including the logs for Key Vault. A little bit of Kql-Fu produced this query:
AzureDiagnostics
| where ResourceProvider =="MICROSOFT.KEYVAULT"
| where ResourceId contains "nameOfVault"
| project CallerIPAddress, OperationName, requestUri_s, ResultType, identity_claim_xms_mirid_sThe resulting columns were:
- CallerIPAddress: The IP address of the client (the IP address used by the App Service Plan VNet integration, in our case)
- OperationName: Things like SecretGet, Authentication, VaultGet, and SecretList
- requestUri_s: The URI of the secret being requested
- ResultType: Was it a success or not?
- identity_claim_xms_mirid_s: The resource ID of the requesting client (the resource ID of the App Service, in our case)
Armed with the resulting info, the dev got what they needed to prove that the App Service was connecting to the Key Vault.
