Amazon.Runtime.AmazonClientException: The retrieved credentials have already expired

[grietine]

Describe the bug

We run ECS on EC2. During the weekend we stop the ECS service and the underlying instance. From time to time we get into situation where the ECS Task can't get the credentials and thus execution of the intended action / call fails. If we restart the ECS Task the issue no longer persists and it gets the credentials. Not sure if it's a bug from AWS SDK side or the underlying services themselves.

Exception:

Amazon.Runtime.AmazonClientException: The retrieved credentials have already expired: Now = 05/19/2025 06:43:19, Credentials expiration = 05/16/2025 23:49:10 at
Amazon.Runtime.RefreshingAWSCredentials.UpdateToGeneratedCredentials(CredentialsRefreshState state, TimeSpan preemptExpiryTime) at
Amazon.Runtime.RefreshingAWSCredentials.GetCredentialsAsync() at
Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext) at
Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at
Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at
Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at
Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at
Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at
Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext) at

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Credentials are refreshed.

Current Behavior

Seems that cached version of credentials (especially at the EC2 / ECS startup) are used.

Reproduction Steps

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

"AWSSDK.Core" Version="3.7.402.5"

Targeted .NET Platform

.NET 9

Operating System and version

AmazonLinux

[dscpinheiro]

The SDK does return cached credentials (even if they're expired) by design - this is for scenarios where the IMDS server is experiencing an outage for example. But even then it should try to refresh them in the background for the next service call.

Is that not the behavior you're seeing? If not, can you share more details no your application (reproduction code would be best)? You can also enable debug logging to see what the SDK is doing:

AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Console;
AWSConfigs.LoggingConfig.LogMetricsFormat = LogMetricsFormatOption.JSON;
AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.Always;
AWSConfigs.LoggingConfig.LogMetrics = true;

var s3 = new AmazonS3Client();
var response = await s3.ListBucketsAsync();

[grietine]

Unfortunately, it's not easily reproducible from my side 😢 At first I thought that the singleton might be the culprit (as both IAmazonKeyManagementService and our wrapper service are registered as singletons), but by looking into AWS SDK code it seems that the background refresh should work.

This error appears semi-regularly, once or twice a month, but in May it was more than 5 and that's the reason I'm contacting you as something is off. I'll try to check if I can get more details the next time it happens.

There was a case when the ECS task was not restarted and it didn't refresh the credentials on many calls for this method, time wise it took ~2h (during which it didn't refresh the credentials successfully) until task was restarted and everything was all good. For other times, I can't 100% say if it refreshes the credentials by itself in couple of minutes (might be the case as well) which still doesn't sound right or we restart the ECS task. At least in AWS Health notifications I didn't see any "IMDS server is experiencing an outage" during those times, but will try to check it as well.

// Registration:
serviceCollection.AddDefaultAWSOptions(configuration.GetAWSOptions());
serviceCollection.AddAWSService<IAmazonKeyManagementService>();

// Usage:
public async Task<CryptographyDataKey> DecryptDataKey(byte[] encryptedKey, CancellationToken cancellationToken = default)
{
	await using var memoryStream = new MemoryStream(encryptedKey);

	var request = new DecryptRequest { CiphertextBlob = memoryStream };

	var response = await _amazonKeyManagementService.DecryptAsync(request, cancellationToken); // <- Here we get the expired credentials exception

	var dataKey = new CryptographyDataKey
	{
		EncryptedKey = encryptedKey,
		PlainKey = response.Plaintext.ToArray()
	};

	return dataKey;
}