Skip to content

[Flaky Test] github.com/elastic/elastic-agent/testing/integration/ess TestFleetManagedUpgradeRollback #16018

Description

Flaky Test

Details

{
  "id": "49665547-a26b-8102-8446-c905bd119fb6",
  "web_url": "https://buildkite.com/organizations/elastic/analytics/suites/elastic-agent-ci/tests/49665547-a26b-8102-8446-c905bd119fb6",
  "scope": "github.com/elastic/elastic-agent/testing/integration/ess",
  "name": "TestFleetManagedUpgradeRollback",
  "location": null,
  "file_name": null,
  "instances": 1,
  "latest_occurrence_at": "2026-08-04T19:24:59.133Z",
  "most_recent_instance_at": "2026-08-04T19:24:59.133Z",
  "last_resolved_at": null,
  "ownership_team_ids": [],
  "failure_examples_count": 1
}

Failure Examples

Example 1:
Run: https://api.buildkite.com/v2/analytics/organizations/elastic/suites/elastic-agent-ci/runs/c2b22bbd-38f0-887a-96a5-0aae02978a17
Time: 2026-08-04T17:27:12.766Z
Stacktrace:

=== RUN   TestFleetManagedUpgradeRollback
    fetcher.go:103: Using existing artifact elastic-agent-9.6.0-SNAPSHOT-linux-x86_64.tar.gz
    fixture.go:438: Extracting artifact elastic-agent-9.6.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestFleetManagedUpgradeRollback3848618208
    fixture.go:456: Completed extraction of artifact elastic-agent-9.6.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestFleetManagedUpgradeRollback3848618208
    fixture.go:1258: Components were not modified from the fetched artifact
    fixture.go:1034: >> running binary with: [/tmp/TestFleetManagedUpgradeRollback3848618208/elastic-agent-9.6.0-SNAPSHOT-linux-x86_64/elastic-agent version --binary-only --yaml]
    fixture.go:1034: >> running binary with: [/tmp/TestFleetManagedUpgradeRollback3848618208/elastic-agent-9.6.0-SNAPSHOT-linux-x86_64/elastic-agent version --binary-only --yaml]
    fetcher_artifact.go:252: Downloading artifact from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.4.4-linux-x86_64.tar.gz
    fetcher_artifact.go:331: Downloading artifact progress 60.85%
    fetcher_artifact.go:331: Downloading artifact progress 100.00%
    fetcher_artifact.go:331: Downloading artifact progress 100.00%
    fetcher_artifact.go:298: Completed downloading artifact from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.4.4-linux-x86_64.tar.gz
    fetcher_artifact.go:252: Downloading artifact from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.4.4-linux-x86_64.tar.gz.sha512
    fetcher_artifact.go:298: Completed downloading artifact from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.4.4-linux-x86_64.tar.gz.sha512
    fetcher_artifact.go:252: Downloading artifact from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.4.4-linux-x86_64.tar.gz.asc
    fetcher_artifact.go:298: Completed downloading artifact from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.4.4-linux-x86_64.tar.gz.asc
    fixture.go:438: Extracting artifact elastic-agent-9.4.4-linux-x86_64.tar.gz to /tmp/TestFleetManagedUpgradeRollback3035239959
    fixture.go:456: Completed extraction of artifact elastic-agent-9.4.4-linux-x86_64.tar.gz to /tmp/TestFleetManagedUpgradeRollback3035239959
    fixture.go:1258: Components were not modified from the fetched artifact
    fixture.go:1034: >> running binary with: [/tmp/TestFleetManagedUpgradeRollback3035239959/elastic-agent-9.4.4-linux-x86_64/elastic-agent version --binary-only --yaml]
    upgrade_fleet_test.go:412: Creating Agent policy...
    upgrade_fleet_test.go:419: Creating Agent enrollment API key...
    upgrade_fleet_test.go:428: Getting default Fleet Server URL...
    upgrade_fleet_test.go:435: Installing Elastic Agent (unprivileged: false)...
    fixture_install.go:201: [test TestFleetManagedUpgradeRollback] Inside fixture install function
    fixture_install.go:229: [test TestFleetManagedUpgradeRollback] Inside fixture installNoPkgManager function
    fixture_install.go:260: >> running binary with: [/tmp/TestFleetManagedUpgradeRollback3848618208/elastic-agent-9.6.0-SNAPSHOT-linux-x86_64/elastic-agent install --force --non-interactive --url https://9721fc73a91a01f70e2381f7a1b309bf.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token TGhRYXpwOEI5dC12Ums2bDIyNjA6dUlsT0ZqQllrUHF6SkZNWTJWdThkdw== --checkin-on-state-change]
    upgrade_fleet_test.go:450: install start agent output:
        Installing in non-interactive mode.
        
[    ] Copying install files  [0s] 
                                   

[    ] Copying install files  [0s] 
                                   

[  ==] Copying install files  [0s] 
                                   

[ ===] Copying install files  [0s] 
                                   

[====] Copying install files  [0s] 
                                   

[=== ] Copying install files  [0s] 
                                   

[==  ] Copying install files  [0s] 
                                   

[=   ] Copying install files  [0s] 
                                   

[    ] Copying install files  [0s] 
                                   

[   =] Copying install files  [0s] 
                                   

[   =] Copying install files  [0s] 
                                   

[ ===] Copying install files  [1s] 
                                   

[====] Copying install files  [1s] 
                                   

[=== ] Copying install files  [1s] 
                                   

[==  ] Copying install files  [1s] 
                                   

[=   ] Copying install files  [1s] 
                                   

[=   ] Copying install files  [1s] 
                                   

[   =] Copying install files  [1s] 
                                   

[  ==] Copying install files  [1s] 
                                   

[  ==] Copying install files  [1s] 
                                   

[====] Copying install files  [2s] 
                                   

[=== ] Copying install files  [2s] 
                                   

[==  ] Copying install files  [2s] 
                                   

[=   ] Copying install files  [2s] 
                                   

[    ] Copying install files  [2s] 
                                   

[   =] Copying install files  [2s] 
                                   

[   =] Copying install files  [2s] 
                                   

[  ==] Successfully copied files  [2s] 
                                       

[  ==] Installing service  [2s] 
                                       

[ ===] Installing service  [2s] 
                                       

[====] Installing service  [2s] 
                                       

[=== ] Installing service  [2s] 
                                       

[==  ] Installing service  [3s] 
                                       

[=   ] Installing service  [3s] 
                                       

[=   ] Installed service  [3s] 
                                       

[=   ] Starting Service  [3s] 
                                       

[    ] Starting Service  [3s] 
                                       

[    ] Service Started  [3s] Elastic Agent successfully installed, starting enrollment.
        
                                       

[    ] Enrolling Elastic Agent with Fleet  [3s] 
                                                

[    ] Waiting For Enroll...  [3s] 
                                                

[    ] Waiting For Enroll...  [3s] 
                                                

[  ==] Waiting For Enroll...  [3s] 
                                                

[ ===] Waiting For Enroll...  [3s] 
                                                

[====] Waiting For Enroll...  [3s] 
                                                

[====] Waiting For Enroll...  [3s] 
                                                

[==  ] Waiting For Enroll...  [3s] 
                                                

[=   ] Waiting For Enroll...  [3s] 
                                                

[    ] Waiting For Enroll...  [4s] {"log.level":"info","@timestamp":"2026-08-04T18:48:20.263Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/enroll.EnrollWithBackoff","file.name":"enroll/enroll.go","file.line":87},"message":"Starting enrollment to URL: https://9721fc73a91a01f70e2381f7a1b309bf.fleet.us-west2.gcp.elastic-cloud.com:443/","ecs.version":"1.6.0"}
        
                                                

[   =] Waiting For Enroll...  [4s] 
                                                

[  ==] Waiting For Enroll...  [4s] 
                                                

[ ===] Waiting For Enroll...  [4s] 
                                                

[====] Waiting For Enroll...  [4s] 
                                                

[=== ] Waiting For Enroll...  [4s] 
                                                

[==  ] Waiting For Enroll...  [4s] 
                                                

[=   ] Waiting For Enroll...  [4s] 
                                                

[    ] Waiting For Enroll...  [4s] 
                                                

[   =] Waiting For Enroll...  [4s] 
                                                

[   =] Waiting For Enroll...  [4s] 
                                                

[ ===] Waiting For Enroll...  [5s] 
                                                

[====] Waiting For Enroll...  [5s] 
                                                

[=== ] Waiting For Enroll...  [5s] {"log.level":"info","@timestamp":"2026-08-04T18:48:21.527Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":396},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2026-08-04T18:48:21.529Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":206},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
        Successfully enrolled the Elastic Agent.
        
                                                

[=== ] Enroll Completed  [5s] 
                                                

[=== ] Done  [5s] 
                                                

[=== ] Done  [5s] 
        Elastic Agent has been successfully installed.
    upgrade_fleet_test.go:460: Setting custom watcher config
    upgrade_fleet_test.go:464: Waiting for Agent to be correct version and healthy...
    fixture.go:985: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    fixture.go:985: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    upgrade_fleet_test.go:474: Agent ID: "d9a869a0-6d52-4c35-88c1-2a9858d84010"
    upgrade_fleet_test.go:476: Waiting for enrolled Agent status to be online...
    upgrade_fleet_test.go:489: Upgrading from version "9.6.0-SNAPSHOT-9b6a62bd1bc87de0a6246a19928b63892bf1d0da" to version "9.4.4-baed89504b1a1c0a7642d60dc0b6c1b0eb34f100"...
    upgrade_fleet_test.go:497: Waiting from upgrade details to show up in Fleet
    upgrade_fleet_test.go:513: Waiting for upgrade watcher to start...
    upgrade_fleet_test.go:518: Upgrade watcher started
    fixture.go:985: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    upgrade_fleet_test.go:532: Waiting for upgraded Agent status to be online...
    upgrade_fleet_test.go:547: Getting Agent version...
    upgrade_fleet_test.go:559: Waiting for upgrade watcher to finish...
    upgrade_rollback_test.go:397: 
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-gcp-1785867140289769265/elastic/elastic-agent/testing/integration/ess/upgrade_rollback_test.go:397
        	Error:      	Received unexpected error:
        	            	waiting for upgrade watcher to finish: failed to fetch watcher pids: failed to fetch pids: FillPidMetrics failed for PID 46792: error getting FD metrics for pid 46792: error reading FD directory for pid 46792: open /proc/46792/fd: no such file or directory
        	Test:       	TestFleetManagedUpgradeRollback
        	Messages:   	managed upgrade failed
    fixture_install.go:325: [test TestFleetManagedUpgradeRollback] Inside fixture cleanup function
    fixture_install.go:341: collecting diagnostics; test failed
    fixture_install.go:833: >> running binary with: [/opt/Elastic/Agent/elastic-agent diagnostics -f /opt/buildkite-agent/builds/bk-agent-prod-gcp-1785867140289769265/elastic/elastic-agent/build/diagnostics/TestFleetManagedUpgradeRollback-2026-08-04T18-49-46Z-diagnostics.zip]
    fixture_install.go:784: >> running binary with: [/opt/Elastic/Agent/elastic-agent uninstall --force]
    fixture.go:1389: Dumping running processes in /opt/buildkite-agent/builds/bk-agent-prod-gcp-1785867140289769265/elastic/elastic-agent/build/diagnostics/TestFleetManagedUpgradeRollback-2026-08-04T18-49-46Z-ProcessDump-cleanup.json
    fixture.go:1708: Temporary directory "/tmp/TestFleetManagedUpgradeRollback3035239959" preserved for investigation/debugging
    fixture.go:1708: Temporary directory "/tmp/TestFleetManagedUpgradeRollback3848618208" preserved for investigation/debugging
--- FAIL: TestFleetManagedUpgradeRollback (120.11s)

Metadata

Metadata

Assignees

No one assigned

    Labels

    flaky-testUnstable or unreliable test cases.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions