Workload Identity with Azure AKS doesn't work as expected

[Ramesh7]

Summary

We are trying to run the Scalr agent pool agent-k8s with Azure AKS cluster but the plan pod is unable to get the identity.

Error

Hi Team,
We are trying to deploy the scalr-agent pools on the AKS cluster, we were successfully able to deploy the helm chart and observed that pods are spinning up as expected. But we are encountering the issue while finding the identity while running plan or apply.
Below is error we have encountered :

Planning failed. Terraform encountered an error while generating this plan.
╷
│
 
Error: 
building account: could not acquire access token to parse claims: ManagedIdentityAuthorizer: failed to request token from metadata endpoint: recei
ved HTTP status 400 with body: {"error":"invalid_request","error_description":"Identity not found"}
│ 
│   with provider["registry.terraform.io/hashicorp/azurerm"],
│   on main.tf line 1, in provider "azurerm":
│    1: provider "azurerm" {
│ 
╵
╷
│
 
Error: 
building account: could not acquire access token to parse claims: ManagedIdentityAuthorizer: failed to request token from metadata endpoint: recei
ved HTTP status 400 with body: {"error":"invalid_request","error_description":"Identity not found"}
│ 
│   with provider["registry.terraform.io/hashicorp/azurerm"].deploy,
│   on main.tf line 6, in provider "azurerm":
│    6: provider "azurerm" {
│ 
╵

Helm Values

Below are the Helm values which we have used :

image:
  repository: scalr/agent
  pullPolicy: Always
  tag: "0.60.0"
agent:
  url: "https://xxxxxx-saas.scalr.io"
  token: "xxxxxxxmasked-tokenxxxxxxxx"
  debug: true
  automount_service_account_token: true
controllerNodeSelector:
  kubernetes.azure.com/agentpool: scalr
workerNodeSelector:
  kubernetes.azure.com/agentpool: scalr
serviceAccount:
  create: true
  name: "scalr-agent-agent-k8s"
  annotations:
    azure.workload.identity/client-id: "xxxxxxxmasked-client-idxxxxxxxxx"
podAnnotations:
  azure.workload.identity/use: "true"
controllerTolerations:
  - key: workload
    operator: Equal
    value: scalr
    effect: NoSchedule
workerTolerations:
  - key: workload
    operator: Equal
    value: scalr
    effect: NoSchedule
extraEnv:
  AZURE_CLIENT_ID: "xxxxxxxmasked-client-idxxxxxxxxx"
  AZURE_TENANT_ID: "xxxxxxxmasked-tenant-idxxxxxxxxx"
  AZURE_FEDERATED_TOKEN_FILE: "/var/run/secrets/azure/tokens/azure-identity-token"

Provider Config

We have tried with and without OIDC in provider but the error doesn't change for us.

provider "azurerm" {
  use_oidc = true # with and without
}