[BUG]: Since May 6th I started to see thousand of warning messages

[gugaiz]

Module Version(s)

1.3.1 and 1.6.1

Bug Report

Since May 6th 11:36 AM EST I started to see these messages on my logs:

2025/05/06 11:36:36 [error] 2494788#2494790: nginx-datadog: Unexpected Remote Configuration status 500 with body (if any, starts on next line):
rpc error: code = Unknown desc = empty targets meta in director local store

Since then, that generated 4.21M logs that were indexed until I realized that those messages were flooding my logs. Why is that generated, can that be fixed?

I am using the following rule to avoid indexing those logs for the moment:

source:nginx (rpc\ error\:\ code\ =\ Unknown\ desc* OR *Unexpected\ Remote\ Configuration\ status\ 500*) 

Reproduction Code

No response

Error Logs

No response

Operating System

No response

[dmehala]

Hi @gugaiz,

The log you're seeing is emitted during the Remote Configuration (RC) process. We understand that this message can be quite verbose and disruptive at scale. We’ll addressed this in the next version of the module, where this specific message will no longer be logged. Apologies for the inconvenience.

If you're not actively using RC, you can disable the feature with the following environment variable:

DD_REMOTE_CONFIGURATION_ENABLED=false

Regarding the volume of logs (4.21M), that seems a lot to me. If this has resulted in unexpected usage or costs, I'll recommend (wearing my non-Datadog employee hat) reaching out to our Support team via this link. They’ll be best positioned to investigate and assist you further.

Please let me know if you need any help disabling the feature or anything else in the meantime.