Windows agent gets unhealthy temporarily on switching from policy with Elastic Defend to Policy without defend.

[amolnater-qasource]

Kibana Build details:

VERSION: 9.1.1 BC1
BUILD: 88427
COMMIT: e3b2d1cf5b2141407d44284364b6d6bcd10634cb

Artifact Link: https://staging.elastic.co/9.1.1-6a4be6bc/downloads/beats/elastic-agent/elastic-agent-9.1.1-windows-x86_64.zip

Host OS:
Windows Server 2025

• Not reproducible on linux platform.

Preconditions:

1. A Windows agent installed and healthy under a policy with Elastic Defend integration enabled.
2. A second policy available without Elastic Defend integration.

Steps to reproduce:

1. Install a Windows agent in a policy with Elastic Defend integration.
2. Switch the agent assignment to a policy that does not include Elastic Defend.
3. Observe that the windows agent temporarily gets unhealthy.

NOTE:

• Issue is observed consistently when switching policies.
• No issue is observed when agent remains on a policy with Defend integration.

Expected Result:
The Windows agent should remain healthy after switching to a policy without Elastic Defend integration.

Screen Capture:

Logs:
elastic-agent-diagnostics-2025-08-06T10-57-37Z-00.zip

Agent JSON:
EC2AMAZ-QKEQ64N-agent-details (3).zip

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[amolnater-qasource]

Suggestion to address the issue:
Version:
Copilot- GPT-4.1

Steps to Modify:

1. Review Policy Change Logic

• File: internal/pkg/agent/application/policy.go
  (or similar agent policy application logic file in the repo)
• Action:
  • Ensure that when the policy is switched and the Defend integration is removed, all related processes, drivers, and configurations are stopped/removed before reporting agent health.
  • Add extra health checks after Defend uninstall to prevent transient unhealthy state.

2. Add Robust Uninstall/Cleanup for Defend Integration

• File: internal/pkg/agent/integrations/defend.go
  (or wherever Elastic Defend uninstall logic is handled)
• Action:
  • Make sure the Defend uninstall process gracefully stops all Windows services, drivers, and cleans up registry/config files.
  • Add logging for each cleanup step.

3. Improve Agent Health Check After Policy Switch

• File: internal/pkg/agent/health/health.go
  (or similar health reporting code)
• Action:
  • Add logic to delay health reporting until after all integration uninstall tasks are finished.
  • Ensure agent does not report "unhealthy" solely due to in-progress uninstall.

4. Add Unit/Integration Tests

• File: internal/pkg/agent/application/policy_test.go
  (or wherever policy change tests are located)
• Action:
  • Add tests to simulate switching from a Defend-enabled policy to a Defend-less policy on Windows.
  • Assert agent remains healthy post-switch.

---

[amolnater-qasource]

@muskangulati-qasource Please review.

[muskangulati-qasource]

Secondary review is Done for this ticket!

[cmacknz]

It looks like it was endpoint that was having problems:

{"log.level":"error","@timestamp":"2025-07-22T07:05:07.469Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.executeCommand.func2","file.name":"runtime/service_command.go","file.line":68},"message":"2025-07-22 07:05:07: debug: ProcFile.cpp:855 Found 12 cgroups for pid(744007)","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-07-22T07:05:07.469Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.executeCommand.func2","file.name":"runtime/service_command.go","file.line":68},"message":"2025-07-22 07:05:07: debug: ProcFile.cpp:861 cgroup: id=12 type=pids path=/system.slice/elastic-agent.service","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-07-22T07:05:07.469Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.executeCommand.func2","file.name":"runtime/service_command.go","file.line":68},"message":"2025-07-22 07:05:07: info: MainPosix.cpp:389 Verifying existing installation","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-07-22T07:05:07.469Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.executeCommand.func2","file.name":"runtime/service_command.go","file.line":68},"message":"2025-07-22 07:05:07: info: InstallLib.cpp:610 Running [/opt/Elastic/Endpoint/elastic-endpoint] [version --log stdout]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-07-22T07:05:07.714Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.executeCommand.func2","file.name":"runtime/service_command.go","file.line":68},"message":"2025-07-22 07:05:07: debug: Exec.cpp:189 ChildMonitor is pid 744010 and monitoring pids 744007 and 744008","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-07-22T07:05:07.820Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.executeCommand.func2","file.name":"runtime/service_command.go","file.line":68},"message":"2025-07-22 07:05:07: info: InstallLib.cpp:650 Installed endpoint is expected version (version: 8.17.2, compiled: Thu Feb 6 02:00:00 2025, branch: HEAD, commit: 72f40668ce48bcd9911ff7d402cad2f5c807d944)","context":"command output","ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2025-07-22T07:05:37.822Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":645},"message":"Component state changed endpoint-default (STARTING->DEGRADED): Degraded: endpoint service missed 1 check-in","log":{"source":"elastic-agent"},"component":{"id":"endpoint-default","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-07-22T07:06:37.822Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":645},"message":"Component state changed endpoint-default (DEGRADED->FAILED): Failed: endpoint service missed 3 check-ins","log":{"source":"elastic-agent"},"component":{"id":"endpoint-default","state":"FAILED","old_state":"DEGRADED"},"ecs.version":"1.6.0"}

The endpoint logs are full of:

{"@timestamp":"2025-07-24T09:58:32.682717002Z","agent":{"id":"ef38519b-48d7-40e9-a847-0919ef2759f1","type":"endpoint"},"ecs":{"version":"8.10.0"},"log":{"level":"warning","origin":{"file":{"line":484,"name":"AgentContext.cpp"}}},"message":"AgentContext.cpp:484 Endpoint is setting status to DEGRADED, reason: Policy Application Status","process":{"pid":3085732,"thread":{"id":780277}}}

This is an X86 RedHat Linux machine:

      host:
        architecture: x86_64
        id: 50f65d38bc4d4f6da40c693a731a78c2
        ip:
            - 127.0.0.1/8
            - 10.80.114.141/24
            - ::1/128
            - fe80::250:56ff:fe8a:e38a/64
        mac:
            - 00:50:56:8a:e3:8a
        name: tvinsta01.bs.ch
        os_family: redhat
        os_platform: rhel
        os_version: 8.10 (Ootpa)
        platform: linux

@elastic/sec-linux-platform can someone take a look at the endpoint logs and see if they can tell what went wrong? It looks like we started endpoint and expected it to connect but it spent some time not being able to talk to agent.