[CI] DataStreamLifecycleDownsampleDisruptionIT testDataStreamLifecycleDownsampleRollingRestart failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #9416 / ubuntu-2404-aarch64_checkpart2_platform-support-arm
• elasticsearch-intake #25650 / part2

Reproduction Line:

./gradlew ":x-pack:plugin:downsample:internalClusterTest" --tests "org.elasticsearch.xpack.downsample.DataStreamLifecycleDownsampleDisruptionIT.testDataStreamLifecycleDownsampleRollingRestart" -Dtests.seed=419D05FAA63230A1 -Dtests.locale=sd-Arab-PK -Dtests.timezone=America/Santarem -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: null

Issue Reasons:

• [main] 2 failures in test testDataStreamLifecycleDownsampleRollingRestart (0.9% fail rate in 217 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testDataStreamLifecycleDownsampleRollingRestart (0.9% fail rate in 217 executions)

Build Scans:

• elasticsearch-periodic-platform-support #9416 / ubuntu-2404-aarch64_checkpart2_platform-support-arm
• elasticsearch-intake #25650 / part2

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nielsbauman]

I don't know if I'll have time to look into this anytime soon, but I'm linking #123769 and its corresponding PR #125478 as we've seen this before. In fact, multiple issues have been opened over time.

[nielsbauman]

FWIW, looking at recent test failures, the main failure still seems to be java.lang.AssertionError: timed out waiting for green state, and not this java.lang.AssertionError: null, which might be a one-off failure.

[nielsbauman]

I looked more into both errors and will open a PR for one of them.

java.lang.AssertionError: timed out waiting for green state: is caused by the relocation of some shards taking more than 30 seconds. This happened after downsampling had already completed, causing the ensureGreen at the end of the test to time out. The distributed team had already suggested disabling shard rebalancing for this test some time ago (among other things, which have already been implemented). I'm opening a PR to disable shard rebalancing in this test class.

java.lang.AssertionError: null: I was only able to find one test failure with this error. Looking at the logs of this test (https://gradle-enterprise.elastic.co/s/uikuaug7fbdda), I see the whole test completed successfully (including the ensureGreen at the end, but somewhere after that, this assertion trips during transport serialization. I'll consider this a one-off failure and/or test framework blip, so I'll leave this. We can tackle it if we do see it occurring more often.