[Bug]:unknown metric: [mem3,shard_cache,hit]

[meshapiro80]

Version

3.5.0

Describe the problem you're encountering

Every day one of my CouchDB servers registers millions of "unknown metric" errors.
Example: "unknown metric: [mem3,shard_cache,hit]"
We do not have any 3rd party monitoring tools.
This is a fresh install of CouchDB 3.5.0 and a single node.

Expected Behaviour

I expect for their not to be millions of these errors as this does not happen on our other CouchDB servers even after a fresh install.

Steps to Reproduce

No response

Your Environment

No response

Additional Context

No response

[rnewson]

hm that hasn't changed in a long time. can you find mem3/priv/stats_descriptions.cfg in your couchdb install and confirm the presence of

{[mem3, shard_cache, hit], [
    {type, counter},
    {desc, <<"number of shard cache hits">>}
]}.

?

[meshapiro80]

Yes, I do see (pasted from my file)

{[mem3, shard_cache, hit], [
    {type, counter},
    {desc, <<"number of shard cache hits">>}
]}.

the path on my system is lib/mem3-3.5.0/priv/stats_descriptions.cfg

[rnewson]

Is it only this one or do your logs name lots of other metrics as missing?

[meshapiro80]

I get many different unknown metric errors:

unknown metric: [couchdb,request_time]
unknown metric: [couch_replicator,connection,acquires]
unknown metric: [couchdb,httpd_status_codes,200]
unknown metric: [couchdb,database_reads]
unknown metric: [couchdb,httpd_request_methods,'GET']
unknown metric: [couchdb,legacy_checksums]
unknown metric: [couchdb,database_writes]

[big-r81]

Which OS do you use?

[big-r81]

Oh, I think I know whats the problem!

TL;DR - Check your file/directory permissions for couchdb directory!
It seems you directory lib/mem3-3.5.0/priv/stats_descriptions.cfg isn't readable by the user who runs CouchDB!

To test this, change the permission of lib/mem3-3.5.0/priv/stats_descriptions.cfg to 644 for example and restart CouchDB.

[meshapiro80]

That would have been a super easy fix!! The OS is Red Hat 8.10 and the permissions of lib all the way down are and have been 755 with the couch user as owner.

[rnewson]

In a remsh (ssh into the machine and run erl -name remsh$$@localhost -remsh couchdb@127.0.0.1 -hidden) try the following commands and share the output please

rp([code:priv_dir(element(1, App)) || App <- application:loaded_applications()]).

and then this one;

rp([file:consult(code:priv_dir(element(1, App)) ++ "/stats_descriptions.cfg") || App <- application:loaded_applications()]).

[nickva]

Does this happen randomly or mostly around when the couchdb is starting or stopping? Before this happens are there any errors in the logs, for instance about couch_stats application crashing or any other exceptions/stacktraces?