Add test cases for RemoteConnector retry behavior

[peternied]

Description

Add test cases for RemoteConnector retry behavior

Refactored product code to make testing easier. Build up these tests to confirm behavior when getting a non-throttled response that was reported by a customer.

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dhrubo-os]

Could you please take updates from main to pass the integ tests. Thanks.

[peternied]

Thanks @dhrubo-os merged from main.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.34%. Comparing base (d7dec0f) to head (f0302bc).
Report is 17 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3504      +/-   ##
============================================
+ Coverage     80.25%   80.34%   +0.09%     
- Complexity     6906     6937      +31     
============================================
  Files           610      610              
  Lines         30077    30262     +185     
  Branches       3368     3384      +16     
============================================
+ Hits          24137    24313     +176     
- Misses         4487     4489       +2     
- Partials       1453     1460       +7     

Flag	Coverage Δ
ml-commons	80.34% <100.00%> (+0.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pyek-bot]

Hi @peternied, just curious, can you please share more context on why this is being done? I see that you mention Build up these tests to confirm behavior when getting a **non-throttled response that was reported by a customer.** Can you share what this report was?

[peternied]

@pyek-bot Sure - I can provide a little more context. Below is a redacted version of their configuration, they setup the client_config policy to retry indefinitely with exponential backoff. However, they never saw any failed requests retried.

Probing in the code base to be sure why this was the case - this looks to be by design. I noticed a lack of test coverage and figured it would be good to area to augment. Especially if someone is interested in changing the behavior so retries occur on a broader set of failed responses.

Redacted configuration

{
  ...
  "parameters": {
    "embeddingTypes": """["float"]""",
    "client_config": """{"retry_timeout_seconds":60,"max_retry_times":-1,"retry_backoff_policy":"exponential_full_jitter"}""",
    ...
  },
  "actions": [
    {
      "action_type": "PREDICT",
      "method": "POST",
      "url": "https://...{REDACTED}...",
      "request_body": """{ "inputText": "${parameters.inputText}", "dimensions": ${parameters.dimensions}, "normalize": ${parameters.normalize}, "embeddingTypes": ${parameters.embeddingTypes} }""",
      ...
    }
  ]
}

[pyek-bot]

Overall change looks good, added some nitpicks. Thanks for the contribution and providing the context!

[peternied]

@pyek-bot @dhrubo-os I've resolved the merge conflict, could I get another review so this change can be merged?

[Zhangxunmt]

The current ITs have covered this change.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-3504-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 dd58f1861c2628ddf1ebad7326f948d554434356
# Push it to GitHub
git push --set-upstream origin backport/backport-3504-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-3504-to-2.x.