[CI] S3RepositoryAnalysisRestIT testRepositoryAnalysis failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #7602 / opensuse-leap-15_platform-support-unix
• elasticsearch-pull-request #67546 / part-2
• elasticsearch-pull-request #67528 / part-2
• elasticsearch-pull-request #67464 / part-2
• elasticsearch-pull-request #67425 / part-2
• elasticsearch-intake #21621 / part2
• elasticsearch-periodic-platform-support #7567 / ubuntu-2004_platform-support-unix
• elasticsearch-pull-request #67190 / part-2
• elasticsearch-pull-request #67191 / part-2
• elasticsearch-pull-request #67180 / part-2

Reproduction Line:

./gradlew ":x-pack:plugin:snapshot-repo-test-kit:qa:s3:javaRestTest" --tests "org.elasticsearch.repositories.blobstore.testkit.analyze.S3RepositoryAnalysisRestIT.testRepositoryAnalysis" -Dtests.seed=B71B1505C30AE167 -Dtests.locale=vi-VN -Dtests.timezone=US/Michigan -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:34247], URI [/_snapshot/repository/_analyze?blob_count=10&seed=-2481787089920117582&max_blob_size=10mb&timeout=120s&concurrency=4], status line [HTTP/1.1 500 Internal Server Error]
{"error":{"root_cause":[{"type":"uncategorized_execution_exception","reason":"Failed execution"}],"type":"repository_verification_exception","reason":"[repository] Elasticsearch observed the storage system underneath this repository behaved incorrectly which indicates it is not suitable for use with Elasticsearch snapshots. Typically this happens when using storage other than AWS S3 which incorrectly claims to be S3-compatible. If so, please report this incompatibility to your storage supplier. Do not report Elasticsearch issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository. See [https://www.elastic.co/docs/api/doc/elasticsearch
[truncated]

Issue Reasons:

• [main] 15 failures in test testRepositoryAnalysis (3.0% fail rate in 505 executions)
• [main] 11 failures in step part-2 (4.6% fail rate in 237 executions)
• [main] 11 failures in pipeline elasticsearch-pull-request (4.7% fail rate in 236 executions)
• [main] 2 failures in pipeline elasticsearch-periodic-platform-support (22.2% fail rate in 9 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 consecutive failures in step part-2
• [main] 2 consecutive failures in pipeline elasticsearch-pull-request
• [main] 9 failures in test testRepositoryAnalysis (2.3% fail rate in 389 executions)
• [main] 7 failures in step part-2 (4.1% fail rate in 172 executions)
• [main] 7 failures in pipeline elasticsearch-pull-request (4.0% fail rate in 175 executions)

Build Scans:

• elasticsearch-periodic-platform-support #7567 / ubuntu-2004_platform-support-unix
• elasticsearch-pull-request #67190 / part-2
• elasticsearch-pull-request #67191 / part-2
• elasticsearch-pull-request #67180 / part-2
• elasticsearch-pull-request #67159 / part-2
• elasticsearch-pull-request #67158 / part-2
• elasticsearch-pull-request #66835 / part-2
• elasticsearch-pull-request #66785 / part-2
• elasticsearch-pull-request #66785 / third-party_s3

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[bcully]

This is bound to be #125737 - I'll look at it tomorrow, I'm rolling onto unplanned anyway by happy chance.

[ywangd]

Thanks for the pointer. IIUC, the copyBlob call like this one needs to fork to a different thread similar to how early read forks.

In the current form, it fails with both the fixture where the S3HttpHandler is single threaded and the real s3 service because the source blob has not finished writing (and abortWrite=false).

It's likely low-risk since I don't think we have this usage pattern in places other than repo analysis. But I am leaving it as medium-risk just in case.

[DaveCTurner]

I think it's a test bug tbh, we can make S3HttpHandler multi-threaded to avoid the deadlock. I like that we wait for the CopyObject to complete (i.e. fail) before sending the last bytes of the upload.