Fleet Server bootstrap is broken on containers; re-enrolls on every container restart

[blakerouse]

When running elastic-agent as a co-located Fleet Server (FLEET_SERVER_ENABLE=1) in a container with the state directory persisted on a volume, the agent re-enrolls on every container restart, producing a new agent record and access API key each time. Previous enrollments become offline ghosts. The persisted state is never honored.

This is a single logical bug with two manifestations, both in shouldFleetEnroll(): the function does not understand the co-located Fleet Server topology and cannot correctly determine whether re-enrollment is needed.

For confirmed bugs, please report:

• Version: 8.19.18+
• Operating System: Containers

[github-actions]

tl;dr: shouldFleetEnroll() currently treats co-located Fleet Server container restarts as a Fleet URL change when FLEET_URL is unset, so it re-enrolls every boot; the fix is to add a Fleet-Server-aware guard in the URL/host comparison path and cover it with a regression test.

Recommendation
Update internal/pkg/agent/cmd/container.go so shouldFleetEnroll() does not require URL/host matching when setupCfg.FleetServer.Enable == true and setupCfg.Fleet.URL == "" (the valid bootstrap case). Keep existing token/API-key validation and enrollment-trigger checks unchanged. Also add a regression test in internal/pkg/agent/cmd/container_test.go for Fleet Server + persisted state + empty URL.

Findings

1. Fleet Server mode enables enrollment even when URL is empty:

// internal/pkg/agent/cmd/setup_config.go:91,99
Enroll: envBool("FLEET_ENROLL", "FLEET_SERVER_ENABLE"),
URL:    envWithDefault("", "FLEET_URL"),

2. Fleet Server bootstrap path does not require --url unless explicitly set:

// internal/pkg/agent/cmd/container.go:507-509,523-526
if cfg.Fleet.URL != "" {
    args = append(args, "--url", cfg.Fleet.URL)
}
...
// URL is only mandatory in non-Fleet-Server enroll path
return nil, errors.New("FLEET_URL is required when FLEET_ENROLL is true without FLEET_SERVER_ENABLE")

3. shouldFleetEnroll() still enforces URL/host matching unconditionally and forces enroll when URL is empty:

// internal/pkg/agent/cmd/container.go:1253-1269
setupFleetHost := setupFleetURL.Host
if setupFleetHost == "" {
    setupFleetHost = setupCfg.Fleet.URL // remains empty when FLEET_URL unset
}
matchedFullURL := slices.Contains(storedFleetHosts, setupCfg.Fleet.URL)
matchedHostOnly := slices.Contains(storedFleetHosts, setupFleetHost)
if !matchedFullURL && !matchedHostOnly {
    return true, nil
}

With persisted state, storedFleetHosts is typically non-empty; empty setup URL/host cannot match, so restart repeatedly re-enters enrollment.

4. Existing tests cover many URL/token permutations but do not cover Fleet Server enablement + empty URL restart behavior (no FleetServer.Enable: true case under TestShouldEnroll):

• internal/pkg/agent/cmd/container_test.go:296-919

5. Related issue/PR lookup was partially blocked by integrity policy for some candidate related issues (#15922, #15408 body fetch denied), so I could not confirm duplicate/precedent discussion from those threads in this run.

Verification

I attempted to run the targeted enrollment decision tests:

go test ./internal/pkg/agent/cmd -run TestShouldEnroll -count=1

Result in this environment:

FAIL github.com/elastic/elastic-agent/internal/pkg/agent/cmd [setup failed]
... reading beats/go.mod: open /home/runner/work/elastic-agent/elastic-agent/beats/go.mod: no such file or directory

So runtime test execution was blocked here by missing submodule checkout, not by test failures in TestShouldEnroll itself.

Detailed Action Plan

1. In internal/pkg/agent/cmd/container.go (shouldFleetEnroll, around 1237-1279), gate URL/host mismatch checks for the Fleet Server bootstrap topology:

   • Detect setupCfg.FleetServer.Enable && setupCfg.Fleet.URL == "".
   • In that branch, skip URL/host/protocol mismatch checks and continue to existing token/hash/API-key validation (1281+, 1319+) so true drift still triggers enrollment.

2. Add regression coverage in internal/pkg/agent/cmd/container_test.go TestShouldEnroll table:

   • Case: cfg.Fleet.Enroll=true, cfg.Fleet.URL="", cfg.FleetServer.Enable=true.
   • Stored config contains non-empty hosts (e.g. (host1/redacted)) and valid API key ACK response.
   • Expected: shouldFleetEnroll == false.

3. Add a second regression case (optional but recommended) for safety:

   • Same Fleet Server + empty URL setup, but invalid API key ACK (client.ErrInvalidAPIKey).
   • Expected: shouldFleetEnroll == true (ensures auth-driven re-enroll still works).

4. Re-run targeted tests once submodules are available:

   • git submodule update --init
   • go test ./internal/pkg/agent/cmd -run TestShouldEnroll -count=1

Related Items

Type	Link / Path	Relevance
Issue	#15971	Current bug report
File	internal/pkg/agent/cmd/setup_config.go:91-99	FLEET_SERVER_ENABLE can imply enroll while URL remains empty
File	internal/pkg/agent/cmd/container.go:507-509,523-526	Fleet Server enroll path treats URL as optional; non-FS path requires URL
File	internal/pkg/agent/cmd/container.go:1253-1269	URL/host mismatch logic that forces re-enroll with empty URL
File	internal/pkg/agent/cmd/container_test.go:296-919	Current TestShouldEnroll table lacks FleetServer-enabled empty-URL restart case
Related issues	#15922, #15408	Found by search, but details not readable in this run due integrity-policy filtering

---

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)