[CI] TimeSeriesDataStreamsIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #7280 / debian-12_platform-support-unix
• elasticsearch-pull-request #64585 / part-5

Reproduction Line:

undefined

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

undefined

Issue Reasons:

• [main] 2 failures in class org.elasticsearch.xpack.ilm.TimeSeriesDataStreamsIT (0.6% fail rate in 357 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nielsbauman]

There are two different failures here. One is about consecutive API calls hitting different nodes with different versions of the cluster state, which I tried to fix in #125621 but will be fixed for real this time in #125815. This failure occurred in the first build scan: https://gradle-enterprise.elastic.co/s/t3rwh5ugyi6du in TimeSeriesDataStreamsIT#testRolloverAction

The issue in the second build scan is caused by ILM being stuck in the wait-for-index-color step - in TimeSeriesDataStreamsIT#testSearchableSnapshotAction. This is exactly the same as in #125683 and #125789. We're currently still discussing the best way to tackle this issue - I opened #125812 for a different test class but we're discussing if we want to go for a different approach.

[nielsbauman]

Since my last comment, #125867 has been opened which addresses the other failure. #125815 addresses the first failure, so I'll let that one close this issue.