Skip to content

Conversation

gaurav137
Copy link
Contributor

@gaurav137 gaurav137 commented Jun 19, 2025

With az cli 2.74 seeing the below failure when using az login --identity with MSI_ENDPOINT set. On debugging the failure figured out that the implementation of fetching the token in az cli has changed and now we have to support the code path that ends up using AZURE_POD_IDENTITY_AUTHORITY_HOST env variable as per https://github.com/AzureAD/microsoft-authentication-library-for-python/blob/d49296c1b2a929a6ab11380e237daa89a5298512/msal/managed_identity.py#L473. This change adds that support.

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.core.windows.net%2F (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7353366e6150>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 193, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/app/main.py", line 296, in login
    return az_cli_ex(args)
           ^^^^^^^^^^^^^^^
  File "/code/app/main.py", line 69, in az_cli_ex
    raise cli.result.error
  File "/usr/local/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 666, in execute
    raise ex
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 734, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 703, in _run_job
    result = cmd_copy(params)
             ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 336, in __call__
    return self.handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 120, in handler
    return op(**command_args)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/command_modules/profile/custom.py", line 158, in login
    return profile.login_with_managed_identity(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/_profile.py", line 286, in login_with_managed_identity
    token = cred.acquire_token(self._arm_scope)[ACCESS_TOKEN]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/auth/msal_credentials.py", line 154, in acquire_token
    result = self._msal_client.acquire_token_for_client(resource=scopes_to_resource(scopes))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/msal/managed_identity.py", line 312, in acquire_token_for_client
    result = _obtain_token(self._http_client, self._managed_identity, resource)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/msal/managed_identity.py", line 431, in _obtain_token
    return _obtain_token_on_azure_vm(http_client, managed_identity, resource)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/msal/managed_identity.py", line 449, in _obtain_token_on_azure_vm
    resp = http_client.get(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/msal/individual_cache.py", line 273, in wrapper
    value = function(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/msal/throttled_http_client.py", line 96, in get
    return NormalizedResponse(self.http_client.get(*args, **kwargs))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.core.windows.net%2F (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7353366e6150>: Failed to establish a new connection: [Errno 111] Connection refused'))

@gaurav137 gaurav137 requested review from a team and Copilot June 19, 2025 12:17
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for az login --identity when using Azure CLI v2.74+ by introducing the AZURE_POD_IDENTITY_AUTHORITY_HOST environment variable and a corresponding metadata token endpoint.

  • Update README with instructions for CLI v2.74+ to use AZURE_POD_IDENTITY_AUTHORITY_HOST
  • Introduce a minimal API route in Program.cs at /metadata/identity/oauth2/token to serve managed identity tokens

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
README.md Added v2.74+ instructions for AZURE_POD_IDENTITY_AUTHORITY_HOST
Program.cs Mapped GET endpoint to serve identity token requests
Comments suppressed due to low confidence (4)

README.md:72

  • [nitpick] Replace the nonstandard '->' bullet with proper markdown blockquote syntax and remove the duplicated instruction already added in line 73.
> [!NOTE]

README.md:73

  • [nitpick] Consolidate the MSI_ENDPOINT note with the v2.74+ instructions to avoid repeating the same sentence twice; consider merging into a single cohesive note block.
> If you are using `az cli` in your service and your service wants to do `az login --identity` then specify `MSI_ENDPOINT`: the URL of the proxy endpoint (e.g., `http://azclicredsproxy:8080/token`) environment variable instead. `IDENTITY_ENDPOINT` and `IMDS_ENDPOINT` are not required for `az login --identity`.

Program.cs:52

  • Consider adding unit or integration tests for this new metadata endpoint to ensure it correctly handles various query parameters and token acquisition scenarios.
app.MapGet("/metadata/identity/oauth2/token", async (HttpContext context, string resource, CancellationToken cancellationToken) =>

Program.cs:58

  • The token response is missing an expires_on field, which many MSAL clients expect; add "expires_on" with the appropriate UNIX timestamp.
        ["expires_in"] = (token.ExpiresOn - DateTimeOffset.UtcNow).TotalSeconds,

Program.cs Outdated
// https://github.com/AzureAD/microsoft-authentication-library-for-python/blob/d49296c1b2a929a6ab11380e237daa89a5298512/msal/managed_identity.py#L473
app.MapGet("/metadata/identity/oauth2/token", async (HttpContext context, string resource, CancellationToken cancellationToken) =>
{
var token = await tokenCredential.GetTokenAsync(new TokenRequestContext([resource]), cancellationToken);
Copy link

Copilot AI Jun 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Using the C# 12 array expression [resource] requires targeting that language version; for broader compatibility, consider new[] { resource }.

Suggested change
var token = await tokenCredential.GetTokenAsync(new TokenRequestContext([resource]), cancellationToken);
var token = await tokenCredential.GetTokenAsync(new TokenRequestContext(new[] { resource }), cancellationToken);

Copilot uses AI. Check for mistakes.

@starcraft66
Copy link
Member

starcraft66 commented Jun 19, 2025

Hey there, I tracked down the bevaviour you're describing to AzureAD/microsoft-authentication-library-for-python#795 which adds support for AAD pod identity to MSAL for python. This appears to be a very old and long-ago-deprecated authentication mechanism. Are you running az cli inside a kubernetes pod with pod identity configured by any chance?

I'm wondering if it is worth supporting pod identity vs just unsetting the AZURE_POD_IDENTITY_AUTHORITY_HOST variable before calling the command line.

@gaurav137
Copy link
Contributor Author

gaurav137 commented Jun 20, 2025

Hi @starcraft66 the scenario is that we have az cli installed within a container running in docker and then in that container we run az login --identity. A simple repro for this situation that you could look at is:

  1. Start az cli container:
    docker run -it mcr.microsoft.com/azure-cli:2.74.0
  2. Run az login --identity in above container, it will fail with a stack trace that contains _obtain_token_on_azure_vm. This took me down the path of using AZURE_POD_IDENTITY_AUTHORITY_HOST env variable.

Now start the azure cli credentials proxy container on a docker network named say credential-proxy-bridge and re-run the az cli container with AZURE_POD_IDENTITY_AUTHORITY_HOST set.
docker run --network credential-proxy-bridge -e AZURE_POD_IDENTITY_AUTHORITY_HOST="http://credentials-proxy:8080" -it mcr.microsoft.com/azure-cli:2.74.0

az login --identity now works with AZURE_POD_IDENTITY_AUTHORITY_HOST set. Previously the MSI_ENDPOINT env variable worked (for which support was added via #47) but we found out that it is no longer working in az cli 2.74.

Thanks for the above pointers. I was not aware this was a deprecated option but then not sure why az cli 2.74 went down that path in the above setup.

@gaurav137
Copy link
Contributor Author

Ping...

@starcraft66
Copy link
Member

Okay, I have had time to do more research on the matter and found the real reason why things broke in azcli 2.74.0: Azure/azure-cli#31577. This old authentication mechanism properly supported using MSI_ENDPOINT to specify the IMDS address. However, the new MSAL for python library they replaced it with did not support that at all. In fact, it was hardcoded to 169.254.169.254... Until when they added support for pod identity. I looked at the code and it seems like a horrible mess of bad detection heuristics to figure out which variables to consider.

Can you try setting the environment variable IDENTITY_HEADER to any value to hopefully convince the library to go down the _obtain_token_on_app_service code path?

@gaurav137
Copy link
Contributor Author

@starcraft66 using IDENTITY_ENDPOINT and IDENTITY_HEADER combination worked as you suggested. Thanks for that. I've updated the PR to update the readme/comments and removed the change I introduced.

@gaurav137
Copy link
Contributor Author

@starcraft66 would you have time to approve the pr?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants