Reduce number of metadata transaction retries

[kfaraz]

Description

Metadata transactions initiated by IndexerSQLMetadataStorageCoordinator and MetadataTaskStorage
typically hold the TaskLockbox.giant lock while the transaction is in progress.
If there is a transient failure, the transaction is retried up to 10 times which typically spans a period of over 5 minutes.

This eventually leads to a very late failure while causing the Overlord threads to be stuck due to the giant lock.

Changes

• Reduce quiet retries to 2
• Reduce max retries to 3. It is best to fail early and noisily. If a retry is needed, it should be done at a higher layer.
• Include retry messages where possible

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[cryptoe]

Changes LGTM. Missed clicking the approve button.

[kfaraz]

Thanks for the review, @cryptoe !