[BUG]: AzureFunctionApp@2 deployment gets stuck and fails sporadically

[david-j-monk]

Task name

AzureFunctionApp

Task version

2.225.1

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

windows-latest

Task log

Starting: AzureFunctionApp
==============================================================================
Task         : Azure Functions Deploy
Description  : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
Version      : 2.225.1
Author       : Microsoft Corporation
Help         : https://aka.ms/azurefunctiontroubleshooting
==============================================================================
Got service connection details for Azure App Service:'***'
Updating App Service Application settings. Data: {"WEBSITES_ENABLE_APP_SERVICE_STORAGE":"true","WEBSITE_RUN_FROM_PACKAGE":"1"}
App Service Application settings are already present.
Package deployment using ZIP Deploy initiated.
##[error]Failed to deploy web package to App Service.
##[warning]Can't find loc string for key: KuduStackTraceURL
##[error]KuduStackTraceURL https://$***:***@***.net/api/vfs/LogFiles/kudu/trace
##[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)
##[warning]Error: Failed to update deployment history. Error: Conflict (CODE: 409)
App Service Application URL: ****
Finishing: AzureFunctionApp

Relevant log output

##[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)

Aditional info

https://github.com/microsoft/azure-pipelines-tasks/issues/14201
Seems to be the same issue as found here

Sometimes takes an hour to fail. If cancelled, sometimes gets the conflict error.

[EEE1992]

Exactly the same error since yesterday

[VictorFavreau]

Having the same error too. All my Azure DevOps / Azure stack is impacted because of this bug I could not deploy. Thanks for your help.

[v-mohithgc]

If possible, can you enable the debug logs (systm.debug=true) and share the logs please?
Thanks

[VpOfEngineering]

Additionally, can you send some dates/times/app names to jzaballos @ microsoft.com? Thank you!

[VictorFavreau]

Following a consultation with Azure Support, the integration of the subsequent parameter into my resource configuration effectively addressed the issue:

WEBSITE_WEBDEPLOY_USE_SCM = "true"

[FinVamp1]

Following a consultation with Azure Support, the integration of the subsequent parameter into my resource configuration effectively addressed the issue:

WEBSITE_WEBDEPLOY_USE_SCM = "true"

@VictorFavreau can you send me the details of your support ticket to finbarr @ microsoft.com so that I can follow up on this. I would like to understand why that setting helped.

[tullydwyer]

We have been experiencing this issue since the start of November.

Switching from this task to Azure CLI seems to have fixed our deployment timing out after 60 mins to <1 minute. No changes to the Function App.

- task: AzureCLI@2
    inputs:
      azureSubscription: "${{ parameters.AzureSubscription }}"
      scriptType: "pscore"
      scriptLocation: "inlineScript"
      inlineScript: |
        az functionapp deployment source config-zip --src "$(Pipeline.Workspace)/artifacts/package.zip" --name "${{ parameters.FunctionAppName }}" --resource-group "${{ parameters.ResourceGroupName }}" --slot "staging"

[eatfrog]

also having this issue, started since a couple of weeks back.
it seems using AzureCLI@2 as suggested above does indeed work. at least the past few deployments.

[LockTar]

We have this as well but not only with functions.

[bernardjeef]

We have the same issue for an Azure Function in a private network with a private endpoint. We are deploying via our custom Windows agent.

I'm using the task :
Task : Azure Functions Deploy
Version : 2.231.0

[LockTar]

@FinVamp1 Have you more information about this problem?

For us a big App Service Plan solved it but we actually don't know why.

WEBSITE_WEBDEPLOY_USE_SCM is by default true right? So why would that help? See documentation for this.

[kardeiz]

+1 to the env var above. For an Azure App Service (running Linux), adding WEBSITE_WEBDEPLOY_USE_SCM=true to the service configuration resolved the issue (the AzureWebApp@1 task was successful).

[FinVamp1]

This is still in our backlog. Plan to introduce a backoff and timeout when calling validation to avoid the task getting stuck.

[santo2]

I'm having this issue now for about a week, didn't find any solution up until now

[olivoarroba]

Hi @FinVamp1, @v-mohithgc, et. al.:

We've been getting this error intermittently for a while now, we've been working with azure support for a while now trying different parameters and configuration settings on the functions deployment without success.

I am attaching a full deployment log of an occurrance of the issue:
DevOps-deployment.txt

Azure support has been pointing to disk space as the root cause of the issue, saying that because we don't have the WEBSITE_RUN_FROM_PACKAGE parameter on our Function app upon creation. Not to paraphrase Azure support conclusion, here it is:

The app is failing to deploy because it is running out of temp storage during deployment.
The app is using temp storage because WEBSITE_RUN_FROM_PACKAGE is not set to 1 before deployment begins, so the site attempts to extract it into temp storage.
WEBSITE_RUN_FROM_PACKAGE is not set to 1 before deployment because it seem to be removed before hand, and then only added back after the deployment begins.

In order to try using the application that way you would need to either use a higher-tier App Service Plan (ASP) or to split the application into several smaller apps. The root cause of the issue we are seeing in the application is that its hitting the limits of temp storage on the consumption plan it's on because it is building without run from package.

However, AzureFunctionAppV2 sets WEBSITE_RUN_FROM_PACKAGE setting before initiating a zipdeploy, as can be seen in the log's line 37940

##[debug]Initiated deployment via kudu service for webapp package : 
Updating App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"}
Deleting App Service Application settings. Data: ["WEBSITE_RUN_FROM_ZIP"]
##[debug]Value of WEBSITE_RUN_FROM_PACKAGE has been changed to 1

The thing that strikes me is that a little bit further down the log in line 37968, the zipdeploy/validate API of KUDU raises a warning that it is recommended to set the setting WEBSITE_RUN_FROM_PACKAGE. Which leads me to think that the setting either was not set correctly, or was set on the wrong slot.

##[debug][POST]https://$dev-device-groups-functions__staging:***@dev-device-groups-functions-staging.scm.azurewebsites.net/api/zipdeploy/validate?&
##[debug]Agent environment resources - Disk: D:\ Available 11532.00 MB out of 14333.00 MB, Memory: Used 2323.00 MB out of 7167.00 MB, CPU: Usage 10.63%
##[debug]loaded affinity cookie ["ARRAffinity=1f50fad77c81e1b5b071b77cac620c6fb65db1f309028733d7c05be6a1c1948a;Path=/;HttpOnly;Secure;Domain=dev-device-groups-functions-staging.scm.azurewebsites.net","ARRAffinitySameSite=1f50fad77c81e1b5b071b77cac620c6fb65db1f309028733d7c05be6a1c1948a;Path=/;HttpOnly;SameSite=None;Secure;Domain=dev-device-groups-functions-staging.scm.azurewebsites.net"]
##[debug]Validation passed response: {"statusCode":200,"statusMessage":"OK","headers":{"content-length":"425","connection":"close","content-type":"application/json; charset=utf-8","date":"Thu, 07 Mar 2024 19:19:46 GMT","server":"Microsoft-IIS/10.0","cache-control":"no-cache","expires":"-1","pragma":"no-cache","set-cookie":["ARRAffinity=1f50fad77c81e1b5b071b77cac620c6fb65db1f309028733d7c05be6a1c1948a;Path=/;HttpOnly;Secure;Domain=dev-device-groups-functions-staging.scm.azurewebsites.net","ARRAffinitySameSite=1f50fad77c81e1b5b071b77cac620c6fb65db1f309028733d7c05be6a1c1948a;Path=/;HttpOnly;SameSite=None;Secure;Domain=dev-device-groups-functions-staging.scm.azurewebsites.net"],"x-ms-request-id":"c863522b-7935-4bca-a95b-a4cb0d8a1368","x-aspnet-version":"4.0.30319","x-powered-by":"ASP.NET"},"body":{"result":"ZipDeploy Validation WARNING: It is recommended to set app setting WEBSITE_RUN_FROM_PACKAGE = 1 unless you are targeting one of the following scenarios: 1. Using portal editing. 2. Running post deployment scripts. 3. Need write permission in wwwroot. 4. Using custom handler with special requirements. NOTE: If you decide to update app setting WEBSITE_RUN_FROM_PACKAGE = 1, you will have to re-deploy your code. "}}
##[warning]"ZipDeploy Validation WARNING: It is recommended to set app setting WEBSITE_RUN_FROM_PACKAGE = 1 unless you are targeting one of the following scenarios: 1. Using portal editing. 2. Running post deployment scripts. 3. Need write permission in wwwroot. 4. Using custom handler with special requirements. NOTE: If you decide to update app setting WEBSITE_RUN_FROM_PACKAGE = 1, you will have to re-deploy your code. "
##[debug]Processed: ##vso[task.issue type=warning;]"ZipDeploy Validation WARNING: It is recommended to set app setting WEBSITE_RUN_FROM_PACKAGE = 1 unless you are targeting one of the following scenarios: 1. Using portal editing. 2. Running post deployment scripts. 3. Need write permission in wwwroot. 4. Using custom handler with special requirements. NOTE: If you decide to update app setting WEBSITE_RUN_FROM_PACKAGE = 1, you will have to re-deploy your code. "
Package deployment using ZIP Deploy initiated.

The command we are using to initiate the deployment via DevOps YAML pipelines is the following:

- task: AzureFunctionApp@2
  displayName: 'Azure Functions App Deploy: staging'
  inputs:
    connectedServiceNameARM: '${{parameters.azureSubscription}}'
    appType: functionApp
    appName: $(appName)
    package: '$(Pipeline.Workspace)/currentBuildArtifact/$(functionAppName)/*.zip'
    deployToSlotOrASE: true
    slotName: staging
    appSettings: >-
      -SOME_VARIABLES $(SOME_VALUES)
    deploymentMethod: 'runFromPackage'

Your attention to this issue is greatly appreciated. I have plenty of occurrences of the issue since we have a couple dozen services in CI/CD pipelines, so if you are in need of more logs, just let me know.

[rochapablo]

The first deploy to a new App Service works, but after that, always getting the error:

##[error]Failed to deploy web package to App Service.
##[warning]Can't find loc string for key: KuduStackTraceURL
##[error]KuduStackTraceURL ***.scm.azurewebsites.net/api/vfs/LogFiles/kudu/trace
##[error]Error: Error: Failed to deploy web package to App Service. Bad Request (CODE: 400)

[TLA020]

Same issue here! Deploying to dev resource works (consumption plan windows)
But when deploying to prod dedicated plan (linux) i get the same error as described above.

After different retries the deployment was successful...

[nubgamerz]

Just want to add that I have the same issue with a zipped package deployment.

It deploys successfully on an automated retry, but it would be nice if it didn't fail every time the first time.

[FinVamp1]

Needs more investigation as the error is the same but the root issue can differ.