Using PaaS (platform-as-a-service) instead of traditional virtual machines running IIS has become a common practice. However, this new world has different challenges than before. A typical design pattern is using your code running in Azure Web Apps and the TLS/SSL being retrieved and maintained by Azure Key Vault. There are other methods available in Azure Web Apps, like exporting the cert (public or private) yourself, having App Service managing the cert, and so on. The Key Vault integration is more elegant for a couple of reasons: first, we can decouple components; second, we can reduce exposure by applying restricted permissions to manage certs at the Key Vault level, and we don’t need to worry about Web App configuration. Another key feature is the ability of a Key Vault to renew the certificate automatically, which reduces a lot the operational cost.
A new emerging pattern is using Azure App Configuration, which acts as a proxy between the application and Azure Key Vault. However, this service does more than that, it allows dynamic settings to be passed to applications, and that service has integration with Key Vault to support secrets. We will cover this service in more detail in an upcoming article here at TechGenix.
This last week, I worked in an incident where the Web Apps were out of sync after an attempt to renew the certificates. The customer has been doing that for years. After switching to Azure, the troubleshooting knowledge from VMs and IIS help, but you need to better understand the integration between Azure services to solve the problem.
Understanding Key Vault out-of-sync error
The customer was facing this error when trying to synchronize: “Failed to sync the certificate: The service does not have access to /subscription/XXX/resourcegroups/XXX/providers/Microsoft.keyvault/vaults/XXX Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation.”
The first step to troubleshoot the issue is to check the TLS/SSL settings of the Web App in question. Logged in the Azure Portal, search for the desired Web App, and then click on TLS/SSL Settings (Item 1), and on the new blade on the right side, click on Private Key Certificates (.pfx) (Item 2).
All certificates configured in the current Web App will be listed (in our case, we can see the health status as yellow, meaning warning). To retrieve more information, click on it (Item 3).
We can trigger a synchronization in the new blade by hitting the button Sync (Item 4). The Web App recycles automatically synchronizes your certificates in 24 hours.
The blade is also essential for troubleshooting. To understand which Key Vault the Web App is using, check the value of the Key Vault Resource URL field (Item 5), the secret name is provided in the field below, and last but not least, we can check the current status of the synchronization. In our case, it is giving OperationNonPermittedOnKeyVault (meaning permissions need some adjustment).
Adjusting Access Policies
One of the items that the cloud administrator must make sure of is that the permissions assigned on the Azure Key Vault are correctly configured, and it can support Web App native integration.
The Microsoft Azure App Service as a platform has the following identity assigned bfa0a7c-a6b6-4736-8310-5855508787cd. Note: If you are from a different Azure Cloud (government and China, to mention a few), check the proper identity that applies to your cloud.
Logged in the Azure Portal, search for a specific Key Vault in the search area, or list all of them by looking at Key Vaults blade. Select the desired Key Vault, click on Access Policies. A list of all existing policies will be displayed, and they will be separated between regular users and applications. Keep in mind that we may have several pages depending on the size of your environment.
Click on Add Access Policy link. In the new blade, select Get permission in the Secret Permissions area (Item 1) and click on None Selected (Item 2). In the new blade, type the application ID related to Microsoft Azure App Service (Item 3) and click on Select (Item 4).
To complete the addition of this new service principal, click on Add (Item 5) and hit save on the next blade that will be displayed.
The final result after hitting the Save button will be depicted in the image below (Item 2).
Some notes from the field
After adjusting the Access Policies in the Vault, you can go back to the Web App and perform the synchronization by hitting the Sync button in the Web App. If you get this error — “The gateway did not receive a response from ‘Microsoft.Web’ within the specified time period” — don’t worry. It could be a timeout to get a response. Most likely, you will see a green status in the certificate, meaning that the synchronization occurred successfully.
Key Vault and Web Apps certificates: Now you know the troubleshooting steps
This article covered the steps required for a cloud administrator to perform the initial troubleshooting steps when having issues synchronizing the certificates between Azure Key Vault and Azure Web Apps.
It is not one size fits all, but it gives you some visibility and opportunities to fix similar issues in your environment before opening a ticket with Microsoft.
If you are using IaC (infrastructure-as-code) — and I hope you are — make sure that you add that access policy entry in your code to avoid issues during the renewal process of your certificate.
Featured image: Shutterstock
