Refactor S3 retries to be in alignment with the AWS SDK. by teo-tsirpanis · Pull Request #5747 · TileDB-Inc/TileDB

teo-tsirpanis · 2026-02-09T15:33:13Z

This PR updates the retry logic of the S3 VFS to be in alignment with the AWS SDK. We added a new config option to set the retry strategy (which defaults to standard), and update the default value of vfs.s3.connect_max_tries to be empty, which lets the AWS SDK pick the value (from environment variables or profile configuration).

Because the AWS SDK's retry strategies don't support setting a scale factor, the vfs.s3.connect_scale_factor config option was deprecated and is no longer supported.

TYPE: FEATURE
DESC: Added vfs.s3.retry_strategy that allows customizing the AWS SDK retry strategy to use for requests to S3. Defaults to standard.

TYPE: BREAKING_BEHAVIOR
DESC: The vfs.s3.connect_scale_factor config option is no longer supported.

TYPE: BREAKING_BEHAVIOR
DESC: The vfs.s3.connect_max_tries config option defaults to an empty string, which lets the AWS SDK determine the maximum number of retries.

Reverts #1651.

ypatia · 2026-02-12T10:58:46Z

LGTM but I'd like @ihnorton to confirm we're ok to do this right now.

ypatia

Let's also add a description as this is a behavior change that can have impact.

ypatia · 2026-02-20T13:59:53Z

tiledb/sm/config/config.cc

 const std::string Config::VFS_S3_CA_FILE = "";
 const std::string Config::VFS_S3_CA_PATH = "";
 const std::string Config::VFS_S3_CONNECT_TIMEOUT_MS = "10800";
-const std::string Config::VFS_S3_CONNECT_MAX_TRIES = "5";


Why remove the default for max tries rather than setting it to 10?

I removed it in order to give the AWS SDK the opportunity to determine the max tries through environment variables and profile settings.

ypatia · 2026-02-20T15:02:42Z

tiledb/sm/filesystem/s3.cc

+  if (s3_params_.connect_max_tries_.has_value()) {
+    retry_strategy = Aws::Client::InitRetryStrategy(
+        s3_params_.connect_max_tries_.value(), s3_params_.retry_strategy_);
+  } else {
+    retry_strategy = Aws::Client::InitRetryStrategy(s3_params_.retry_strategy_);
+  }


Suggested change

if (s3_params_.connect_max_tries_.has_value()) {

retry_strategy = Aws::Client::InitRetryStrategy(

s3_params_.connect_max_tries_.value(), s3_params_.retry_strategy_);

} else {

retry_strategy = Aws::Client::InitRetryStrategy(s3_params_.retry_strategy_);

}

retry_strategy = Aws::Client::InitRetryStrategy(

s3_params_.connect_max_tries_.value_or(10), s3_params_.retry_strategy_);

We can't do that; the overload with the one parameter tries to retrieve the max retries from environment variables and profile configuration.

ypatia · 2026-02-20T15:12:23Z

tiledb/sm/filesystem/s3.h

-      , connect_scale_factor_(config.get<int64_t>(
-            "vfs.s3.connect_scale_factor", Config::must_find))
+      , connect_max_tries_(config.get<int64_t>("vfs.s3.connect_max_tries"))
+      , has_connect_scale_factor_(


the retry_strategy_?

Oops, added.

ypatia · 2026-02-20T15:16:24Z

tiledb/sm/filesystem/s3.cc

+
+  virtual long CalculateDelayBeforeNextRetry(
+      const Aws::Client::AWSError<Aws::Client::CoreErrors>& error,
+      long attemptedRetries) const {


override on the rest of the methods?

ypatia · 2026-02-20T15:21:30Z

tiledb/sm/config/config.cc

-  } else if (param == "vfs.s3.connect_max_tries") {
-    RETURN_NOT_OK(utils::parse::convert(value, &vint64));
-  } else if (param == "vfs.s3.connect_scale_factor") {
+  } else if (param == "vfs.s3.connect_max_tries" && !value.empty()) {


maybe we can add sanity check for the allowed values in retry_strategy as well?

I prefer we don't, and pass the value to the AWS SDK as-is. It can handle invalid values.

ypatia · 2026-02-20T15:24:25Z

tiledb/sm/filesystem/s3.cc

              // SSOCredentialsProvider
              "TooManyRequestsException"},
-          s3_params_.connect_max_tries_);
+          s3_params_.connect_max_tries_.value_or(10));


why not just rely on the parameter default?

We can, but I thought of letting the config option influence retries in authentication requests as well.

teo-tsirpanis · 2026-02-20T16:04:05Z

Updated description.

Always use exponential backoff to retry S3 requests.

920b15b

Reverts #1651.

teo-tsirpanis requested a review from ypatia February 9, 2026 15:33

ypatia requested a review from ihnorton February 12, 2026 10:53

teo-tsirpanis requested a review from a team as a code owner February 19, 2026 01:47

teo-tsirpanis force-pushed the teo/core-495-improve-our-slow_down-retry-strategy branch from 410ffd0 to b6cbe16 Compare February 19, 2026 02:00

teo-tsirpanis added 2 commits February 19, 2026 04:02

Add config option for retry strategy.

9957788

Deprecate vfs.s3.connect_scale_factor.

e57322f

teo-tsirpanis force-pushed the teo/core-495-improve-our-slow_down-retry-strategy branch from b6cbe16 to e57322f Compare February 19, 2026 02:02

Fix test.

aaa6c80

teo-tsirpanis changed the title ~~Always use exponential backoff to retry S3 requests.~~ Refactor S3 retries to be in alignment with the AWS SDK. Feb 19, 2026

ypatia reviewed Feb 20, 2026

View reviewed changes

teo-tsirpanis added 2 commits February 20, 2026 17:52

Set retry strategy to S3Parameters.

1771d09

Override all methods of SlowDownTrackingRetryStrategy.

91820fd

teo-tsirpanis requested a review from ypatia February 20, 2026 16:04

Fix casing of include.

71dc49f

Comments

Conversation

teo-tsirpanis commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ypatia commented Feb 12, 2026

Uh oh!

ypatia left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teo-tsirpanis commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

teo-tsirpanis commented Feb 9, 2026 •

edited

Loading