Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IMDS returns SdkClientException on 4XX errors, but reading autoscaling lifecycle state can return 404. #5786

Closed
2 tasks done
benjumanji opened this issue Jan 9, 2025 · 5 comments
Closed
2 tasks done

IMDS returns SdkClientException on 4XX errors, but reading autoscaling lifecycle state can return 404. #5786

benjumanji opened this issue Jan 9, 2025 · 5 comments
Labels
ec2-metadata-client feature-request A feature should be added or improved. p2 This is a standard priority issue

Comments

@benjumanji
Copy link

benjumanji commented Jan 9, 2025 •
edited
Loading

Describe the feature

IMDS client should return SdkServiceException when an http request is returned from the service. It's use of SdkClientException doesn't match the description in the docs: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/handling-exceptions.html#sdkclientexception, to wit:

An SdkClientException is generally more severe than an SdkServiceException, and indicates a major problem that is preventing the client from making service calls to AWS services

A request that recieves a response cannot be a client exception. It made the call, the service rejected it.

Use Case

IMDS client returns SdkClientExceptions unconditionally on 4XX errors (

aws-sdk-java-v2/core/imds/src/main/java/software/amazon/awssdk/imds/internal/AsyncHttpRequestHelper.java

Line 89 in 7eae4d8

throw SdkClientException.builder().message(responseContent).build();
) but when an instance has been up for some length of time then reading the autoscaling lifecycle state can fail with a 404, this is not a client error and is retryable. We don't want to log these, but the status code isn't accessible to us because the client threw away that information.

Proposed Solution

Return an SdkServiceException any time there is an actual service response.

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

AWS Java SDK version used

2

JDK version used

17

Operating System and version

linux 6.11 (nixos)
@benjumanji benjumanji added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Jan 9, 2025
@debora-ito
Copy link
Member

debora-ito commented Jan 14, 2025 •
edited
Loading

@benjumanji do you have a stacktrace for the specific exception you have in mind?

@debora-ito debora-ito added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 10 days. and removed needs-triage This issue or PR still needs to be triaged. labels Jan 14, 2025
@benjumanji
Copy link
Author

benjumanji commented Jan 21, 2025 •
edited
Loading

Do you mean a stack trace illustrating the question? Or the stack trace I would like to have in the future? Assuming it is the former:

{
  "level": "ERROR",
  "ts": "1736891693048",
  "name": "hypervolt.aws.imds.AwsMetadataClient",
  "message": "Failed to request target state from aws metadata service, will try again in 10 seconds seconds",
  "stacktrace": {
    "class": "software.amazon.awssdk.core.exception.SdkClientException",
    "message": "<?xml version=\"1.0\" encoding=\"iso-8859-1\"?>\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"\n\t\t \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n<html xmlns=\"http://www.w3.org/1999/xhtml\" xml:lang=\"en\" lang=\"en\">\n <head>\n  <title>404 - Not Found</title>\n </head>\n <body>\n  <h1>404 - Not Found</h1>\n </body>\n</html>\n",
    "backtrace": [
      "at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)",
      "at software.amazon.awssdk.imds.internal.AsyncHttpRequestHelper.handleResponse(AsyncHttpRequestHelper.java:89)",
      "at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)",
      "at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)",
      "at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)",
      "at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)",
      "at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)",
      "at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$DataCountingPublisher$1.onComplete(ResponseHandler.java:519)",
      "at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:254)",
      "at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.access$600(ResponseHandler.java:77)",
      "at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onComplete(ResponseHandler.java:375)",
      "at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.publishMessage(HandlerPublisher.java:402)",
      "at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.flushBuffer(HandlerPublisher.java:338)",
      "at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.receivedDemand(HandlerPublisher.java:291)",
      "at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.access$200(HandlerPublisher.java:61)",
      "at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:495)",
      "at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)",
      "at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)",
      "at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)",
      "at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)",
      "at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)",
      "at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)",
      "at java.base/java.lang.Thread.run(Thread.java:833)",
      "at delay @ hypervolt.aws.imds.AwsMetadataClient.readMetadata(AwsMetadataClient.scala:47)",
      "at fromCompletableFuture @ hypervolt.aws.imds.AwsMetadataClient.readMetadata(AwsMetadataClient.scala:47)",
      "at map @ hypervolt.aws.imds.AwsMetadataClient.readMetadata(AwsMetadataClient.scala:47)"
    ],
    "cause": null
  }
}

Note that this is an SdkClientException so there is no status code member to interrogate, the best we can do is string matching on the message which is objectively bad.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 10 days. label Jan 21, 2025
@debora-ito debora-ito added needs-triage This issue or PR still needs to be triaged. needs-review This issue or PR needs review from the team. and removed needs-triage This issue or PR still needs to be triaged. labels Jan 21, 2025
@debora-ito debora-ito self-assigned this Jan 27, 2025
@debora-ito
Copy link
Member

Thank you for the stacktrace, @benjumanji. Feature request acknowledged.

Community note: please vote by adding a 👍 to the top comment to help us with prioritization.

@debora-ito debora-ito added p2 This is a standard priority issue and removed needs-review This issue or PR needs review from the team. labels Feb 13, 2025
@debora-ito debora-ito removed their assignment Feb 13, 2025
@S-Saranya1 S-Saranya1 mentioned this issue Mar 11, 2025
Merged
12 tasks
@S-Saranya1 S-Saranya1 mentioned this issue Mar 24, 2025
Merged
12 tasks
@S-Saranya1
Copy link
Contributor

Hi @benjumanji,
We've addressed this issue by introducing 'Ec2MetadataClientException' which now provides access to HTTP status codes and error response bodies, providing additional error context from the response. This allows you to implement specific handling for different error types. Please let us know if you need anything else.

@github-actions GitHub Actions
Copy link

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ec2-metadata-client feature-request A feature should be added or improved. p2 This is a standard priority issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@benjumanji @debora-ito @S-Saranya1