metrics-endpoint-relation-created on every unit fail in cluster mode

[erik78se]

I have a juju controller 3 node cluster.

I have a alloy-vm charm which can integrate with the controller via its metrics-endpoint

When I relate it to my monitoring charm "alloy-vm" which is capable of scraping metrics the relation hook ends up in an error state.

juju switch admin/controller juju integrate controller:metrics-endpoint alloy-vm:metrics-endpoint

It ends up in this:

File "/var/lib/juju/agents/unit-controller-0/charm/venv/ops/framework.py", line 947, in _reemit custom_handler(event) File "/var/lib/juju/agents/unit-controller-0/charm/./src/charm.py", line 81, in _on_metrics_endpoint_relation_created self.control_socket.add_metrics_user(username, password) File "/var/lib/juju/agents/unit-controller-0/charm/src/controlsocket.py", line 133, in add_metrics_user resp = self._request( ^^^^^^^^^^^^^^ File "/var/lib/juju/agents/unit-controller-0/charm/src/controlsocket.py", line 80, in _request response = self._request_raw(method, path, query, headers, data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/juju/agents/unit-controller-0/charm/src/controlsocket.py", line 126, in _request_raw raise APIError(body, code, status, message) controlsocket.APIError: user "juju-metrics-r1" already exists unit-controller-0: 18:43:40 ERROR juju.worker.uniter.operation hook "metrics-endpoint-relation-created" (via hook dispatching script: dispatch) failed: exit status 1

[dimaqq]

Reformatted stack trace:

File "/var/lib/juju/agents/unit-controller-0/charm/venv/ops/framework.py", line 947, in _reemit
  custom_handler(event)
File "/var/lib/juju/agents/unit-controller-0/charm/./src/charm.py", line 81, in _on_metrics_endpoint_relation_created
  self.control_socket.add_metrics_user(username, password) 
File "/var/lib/juju/agents/unit-controller-0/charm/src/controlsocket.py", line 133, in add_metrics_user
  resp = self._request(
         ^^^^^^^^^^^^^^ 
File "/var/lib/juju/agents/unit-controller-0/charm/src/controlsocket.py", line 80, in _request
  response = self._request_raw(method, path, query, headers, data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/var/lib/juju/agents/unit-controller-0/charm/src/controlsocket.py", line 126, in _request_raw
  raise APIError(body, code, status, message)
controlsocket.APIError: user "juju-metrics-r1" already exists 

unit-controller-0: 18:43:40 
ERROR juju.worker.uniter.operation hook "metrics-endpoint-relation-created" (via hook dispatching script: dispatch) failed: exit status 1

[dimaqq]

Summary: xx fails with user "juju-metrics-r1" already exists in cluster mode

[erik78se]

@dimaqq will you be able to patch it up?

I will also try get the logs from tje controller into loki eventually.

[dimaqq]

I don't own this codebase today.
What I can do is advocate for Charm Tech to invest some time in the overall implementation/mainenance/testing/ci/etc. in the upcoming cycle.

[dimaqq]

Someone from Juju is better qualified to comment what needs to be done:

• make user creation idempotent (and how to share the password), or
• create separate users (per cluster member?)

[manadart]

This looks to be an issue in Juju itself, rather than the charm. We're failing a check to ensure that this would be an idempotent operation. Can you look at the user and see if it is disabled, or marked as deleted?

[jameinel]

Should we use a single user for all metrics, or should we use a user-per-relation? It seems cleaner that 2 different applications that are monitoring the juju controller would use separate identities for doing the scraping. John =:->

…

On Tue, Apr 7, 2026 at 4:48 AM Joseph Phillips ***@***.***> wrote: *manadart* left a comment (juju/juju-controller#112) <#112 (comment)> This looks to be an issue in Juju itself, rather than the charm. We're failing a check to ensure that this would be an idempotent operation. Can look at the user and see if it is disabled, or marked as deleted? — Reply to this email directly, view it on GitHub <#112 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABRQ7MA5AYQX6JVZF7C3XL4US6GBAVCNFSM6AAAAACXCLEVCWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCOJXG4ZDAMRUGU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[SimonRichardson]

Note: This doesn't currently happen on 4.0.

[erik78se]

Should we use a single user for all metrics, or should we use a
user-per-relation? It seems cleaner that 2 different applications that are
monitoring the juju controller would use separate identities for doing the
scraping.

I would think that a per-relation user would make sense, until the cleanup work begins with lingering users... Perhaps as a first implementation use a single user.

[erik78se]

Any progress on this? We still have a controller charm left in error mode due to this issue. We were hoping to get a fix for this to be able to monitor it properly.