Agents logging constant error: "failed to add secret markers: secret path outputs.OUTPUT_ID.service_token does not exist"

[leandrojmp]

Hello,

We noticed recently that agents are constantly reporting the following error:

failed to add secret markers: secret path outputs.OUTPUT_ID.service_token does not exist

Where the OUTPUT_ID is the remote elasticsearch output configured in the Elastic Agent policy to send monitoring logs and metrics to our monitoring cluster.

This does not seems to completely impact the delivery of logs and metrics as we are seeing these logs in the monitoring cluster.

Checking the output settings using the request: GET kbn:/api/fleet/outputs/OUTPUT_ID, we have this response:

{
  "item": {
    "id": "OUTPUT_ID",
    "name": "es-cloud-monitoring",
    "type": "remote_elasticsearch",
    "hosts": [
      "https://ENDPOINT.aws.found.io:443"
    ],
    "is_default": false,
    "is_default_monitoring": true,
    "preset": "balanced",
    "config_yaml": "",
    "secrets": {
      "service_token": {
        "id": "SERVICE_TOKEN_ID"
      }
    },
    "service_token": null,
    "sync_integrations": false,
    "kibana_url": null,
    "sync_uninstalled_integrations": false,
    "write_to_logs_streams": false,
    "kibana_api_key": null,
    "ssl": {
      "certificate": "",
      "certificate_authorities": []
    }
  }
}

We can see that the field service_token is null, but the field secrets.service_token is populated.

The configuration of the output was done following the instructions in the Fleet UI.

It seems that this issue was introduced between 8.19.2 and 9.4.4, I had an agent on 8.19.2 that was not reporting this error and as soon as I upgrade it to 9.4.4 it started reporting those messages.

On version 8.19.2:

After upgrade to version 9.4.4:

The integrations that I have in the policies that are sending data to the monitoring cluster are:

• Elastic Agent
• System (with only the Collect metrics from System instances option enabled)

To replicate:

I think this can be replicated with the followint steps:

• Create 2 cloud deployments, one will be used for monitoring.
• Add a remote elasticsearch output on the main deployment.
• Create a policy to send metrics and elastic agent logs to the monitoring cluster.
• Add an agent on version 9.4.4 to this policy.
• Validate the error logs on the monitoring cluster.

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[leandrojmp]

I had to downgrade a couple of agents to 8.19.19 to lower the chances of impact on the monitoring of my agents, there is another current Fleet issue on 9.4.X that makes agents not sync policies updates because of version-specific policies introduced on 9.4, so at the moment having some important agents on 8.19.19 is safer.

A couple of agents stopped send monitoring data after a policy change, the only way to get the data back was downgrading the agent.

[leandrojmp]

Hello,

I just identified that this may be also impacting the enrollment of new agents, we have a process where when new instances are created the automation will install and enroll the agent.

New agents enroll on an empty policy without any integration, even this way they are receiving the following message:

[root@host ~]# elastic-agent status
┌─ fleet
│  └─ status: (FAILED) status code: 500, fleet-server returned an error: BadRequest, message: processPolicy: failed to prepare output "OUTPUT_ID": failed to get service token from output: OUTPUT_ID
└─ elastic-agent
   └─ status: (STARTING) Waiting for initial configuration and composable variables

The OUTPUT_ID is the endpoint for the monitoring cluster, which is defined as the default output for monitoring data.

[leandrojmp]

Hello,

I've made some troubleshooting and have some findings.

First, this is what we have in this scenario:

• 2 deployments on Elastic Cloud, one is the logs deployments and the other acts as the monitoring deployment.
• The monitoring deployment receives Elastic Cloud logs from other deployments in the same organization.
• The monitoring deployment receives logs and metrics from some Elastic Agents in specific Agent Policies.
• The monitoring deployment receives system metrics from some hosts using Elastic Agent in specific Agent policies.
• The monitoring deployment is configured as the default monitoring output in the Fleet settings on the logs deployment, it was configured using the Remote Elasticsearch option.
• The specific agent policies had the Collect Agents logs and Collect Agent Metrics options enabled.
• The specific agent policies had an Elastic Agent integration (shared) with the output configured to the monitoring deployment.
• The specific agent policies had an System integration (shared) with the output configured to the monitoring deployment and configured to get only metrics.
• New agents are enrolled on empty onboarding policies without any integration and are moved to their policies using automation.

This was working without any issues up until 9.4.1, or at least we didn't noticed any problem in the monitoring or enrollment of agents.

In 9.4.1 we were (and still are) affected by an issue that has some impact in the syncing of changes on agent policies, some agents were not getting the updates in the policies and were in an outdated policy state.

This issue was partially fixed on 9.4.3 and this week we upgraded our stack on both deployments to 9.4.4 and started upgrading some agents, this is when we started seeing this error and noticed some impact in the enrollment of agents (our image has the version set on 9.4.1)

Our issues were:

• Agents on version 9.4.X were constantly logging the following error:

failed to add secret markers: secret path outputs.OUTPUT_ID.service_token does not exist

I believe this is related to the System integration that is configured to send data to an output configured as remote_elasticsearch, which was until then also the default Agent monitoring output.

• Some agents mostly on version 9.4.X were having issues to enroll in their onboarding policies, they would stay on an Updating state until they went offline, this also happened with agents on earlier versions like 9.3.X and 8.19.X

This is the error I saw on an agent I was configuring:

[root@host ~]# elastic-agent status
┌─ fleet
│  └─ status: (FAILED) status code: 500, fleet-server returned an error: BadRequest, message: processPolicy: failed to prepare output "OUTPUT_ID": failed to get service token from output: OUTPUT_ID
└─ elastic-agent
   └─ status: (STARTING) Waiting for initial configuration and composable variables

I believe this is related to the fact that the default Agent monitoring output was set to an remote_elasticsearch.

Note

Both errors seems to be related to a service token in an remote elasticsearch output.

After some troubleshooting this is what we found out:

• Having an Elastic Agent integration in a policy is redundant, it does nothing, the collecting of logs and metrics seems to be controlled only by the Collect Agents logs and Collect Agent Metrics options in each policy, this will then send data to the output that is marked as the default output for Agent monitoring.
• Information about the default Agent integrations and _ Agent monitoring_ output will be sent to all agents enrolling in the cluster, regardless of which policy they are being enrolled.

And this is what we changed:

• We disabled the Collect Agents logs and Collect Agent Metrics options in all policies that had this enabled, this impacts only the elastic agent process logs and metrics, we still have the System logs and Metrics. (We will reenable this after we change every policy to use a different output instead of the default one.)
• We changed the default Agent monitoring output to be the same as the Agent integration output, since there is no policy getting this monitoring data, this will have no impact, but will help to avoid issues with enrollment.
• We removed the Elastic Agent integrations from all policies is this does nothing, not sure why users can add it on integrations though.

This what we have now:

• No problems on enrollment, agents get into Healthy state as soon as they enroll on fleet.
• Agents on version 9.4.X with the System integration that send metrics to the remote elasticsearch monitoring deployment are constantly logging the same error as before. (verified on 9.4.3 and 9.4.4).
• Agents on version lower than 9.4.X with the same integration as above does not emit the same error. (verified on 9.3.3 and 8.19.X)

[khushijain21]

This is the related issue on fleet-server elastic/fleet-server#5664. I am continuing investigation on this