Trust but verify. Recently, during a project, we encountered an interesting issue: a few Azure Durable Function invocations were stuck in a “running” status, while others (both older and newer) had completed successfully.
After some investigation, we discovered that the status displayed in the Azure portal was misleading. The functions had actually completed, but the status hadn’t refreshed, even after restarting the Function instance multiple times.
Steps to Investigate and Resolve the Issue
I tackled the problem in a few steps:
- Restarting the Function — “Have you tried turning it off and on again?” Yes, multiple times. It didn’t help.
- Searching for ways to terminate the function invocation — Found the right API calls.
- Handling bearer tokens — A bit tricky, but we got it sorted.
- Checking the function’s status — Verification through API.
- Termination (optional) — Preparing to terminate if necessary.
Restarting the Azure Durable Function
This step is pretty basic, and I won’t go into detail here. You can restart the function using the Azure portal, PowerShell, or Azure CLI.
For PowerShell, here are useful links:
Searching for a Way to Terminate the Azure Durable Function Invocation
Thankfully, this part was straightforward. I quickly found Microsoft documentation providing the necessary API calls:
Checking the Azure Durable Function Status via API
Things got a bit trickier here. I needed to obtain a token to authenticate the API calls. I first tried get-AzToken | clip in PowerShell, but it didn’t work. Then I tried grabbing a token from browser developer tools, which also failed.
Eventually, I found a method that worked.
API Call Setup
To make the API calls, I used Thunder Client in Visual Studio Code, though you can use Postman or any other API tool. From the Azure portal, I copied the resource ID of the function invocation I wanted to check.
Authentication
For authentication, I used a PowerShell script from this GitHub repo. This script provided the bearer token necessary for making the API calls.
Here’s a snippet of the PowerShell code that worked (thanks to the Martin Brandl’s code, the link to the GitHub repo is above):
Connect-AzAccount -SubscriptionId <yoursubscriptionID>
$ResourceGroupName = “<yourResourceGroupName>”
$FunctionAppName = “<yourFunctionAppName>”
$publishingCredentials = Invoke-AzResourceAction `
-ResourceGroupName $ResourceGroupName `
-ResourceType 'Microsoft.Web/sites/config' `
-ResourceName ('{0}/publishingcredentials' -f $FunctionAppName) `
-Action list `
-ApiVersion 2019-08-01 `
-Force
$base64Credentials = [Convert]::ToBase64String(
[Text.Encoding]::ASCII.GetBytes(
('{0}:{1}' -f $publishingCredentials.Properties.PublishingUserName, $publishingCredentials.Properties.PublishingPassword)
)
)
Invoke-RestMethod `
-Uri ('https://{0}.scm.azurewebsites.net/api/functions/admin/token' -f $FunctionAppName) `
-Headers @{ Authorization = ('Basic {0}' -f $base64Credentials) }
With the token and resource ID in hand, I could now make the API call to check the function’s status.
Getting the Status
I used the following API call:
GET https://<name>.azurewebsites.net/runtime/webhooks/durabletask/instances/<durableFunctionInstanceID>
To my surprise, I received a 200 OK response, and the status showed “runtimeStatus”: “Completed”. This didn’t match the “running” status I had been seeing in the Azure portal, confirming that the GUI was misleading.
As the saying goes, “Trust but verify”—especially when it comes to GUI statuses.
Termination (Optional)
Before terminating the function, I wanted to ensure it was actually still running by using the Get Status API.
Potential responses you can get include:
- 200 OK: The instance is completed or failed.
- 202 Accepted: The instance is still in progress.
- 400 Bad Request: The instance failed or was terminated.
- 404 Not Found: The instance doesn’t exist or hasn’t started.
- 500 Internal Server Error: The instance failed due to an unhandled exception.
For both status checks and terminations, you need to provide the bearer token in the authentication section.
Terminating the Function
If you need to terminate an instance, use this API call:
POST https://<name>.azurewebsites.net/runtime/webhooks/durabletask/instances/<durableFunctionInstanceID>/terminate?reason=<yourReason>
Provide the instance ID and token just as with the status check. You can also add a reason for the termination, which is useful for future reference.
I hope this article helps anyone encountering a similar issue. It’ll definitely save me time if I need to revisit this process in the future—especially remembering that get-AzToken doesn’t always work!
And for more insights into avoiding common cloud pitfalls, check out my book “Mind The Gap: Most Common Cloud Mistakes“.