EdgeX Foundry is unable to refresh the token in secured mode

[mohammedimrankasab]

We have used the EdgeX Foundry 3.1.1 version, with secure mode enabled docker-compose deployment.

Everything works smoothly until after a couple of weeks when some service complains about the token renewal failure. This failure is mostly seen in the security-proxy-auth container.

When exec inside the container, /tmp/edgex/secrets/security-proxy-auth/secrets-token.json is missing.

logs from the security-proxy-auth

Wed Oct 30 07:52:17 UTC 2024 Starting entrypoint.sh /security-proxy-auth -cp=consul.http://edgex-core-consul:8500 --registry level=INFO ts=2024-10-30T07:52:17.647023387Z app=security-proxy-auth source=secret.go:69 msg="Creating SecretClient"
level=INFO ts=2024-10-30T07:52:17.647262464Z app=security-proxy-auth source=variables.go:462 msg="Variables override of 'SecretStore/Host' by environment variable: SECRETSTORE_HOST=edgex-vault"
level=INFO ts=2024-10-30T07:52:17.647350056Z app=security-proxy-auth source=secret.go:172 msg="SecretStore information created with 1 overrides applied"   
level=INFO ts=2024-10-30T07:52:17.647363368Z app=security-proxy-auth source=secret.go:79 msg="Reading secret store configuration and authentication token"
level=INFO ts=2024-10-30T07:52:17.647394784Z app=security-proxy-auth source=secret.go:216 msg="load token from file"
level=WARN ts=2024-10-30T07:52:17.647497904Z app=security-proxy-auth source=secret.go:132 msg="Retryable failure while creating SecretClient: open /tmp/edgex/secrets/security-proxy-auth/secrets-token.json: no such file or directory"
level=INFO ts=2024-10-30T07:52:18.648176579Z app=security-proxy-auth source=secret.go:79 msg="Reading secret store configuration and authentication token"
level=INFO ts=2024-10-30T07:52:18.648493379Z app=security-proxy-auth source=secret.go:216 msg="load token from file"
level=WARN ts=2024-10-30T07:52:18.648565731Z app=security-proxy-auth source=secret.go:132 msg="Retryable failure while creating SecretClient: open /tmp/edgex/secrets/security-proxy-auth/secrets-token.json: no such file or directory"
level=INFO ts=2024-10-30T07:52:19.648958692Z app=security-proxy-auth source=secret.go:79 msg="Reading secret store configuration and authentication token"
level=INFO ts=2024-10-30T07:52:19.649020629Z app=security-proxy-auth source=secret.go:216 msg="load token from file"
level=WARN ts=2024-10-30T07:52:19.64905984Z app=security-proxy-auth source=secret.go:132 msg="Retryable failure while creating SecretClient: open /tmp/edgex/secrets/security-proxy-auth/secrets-token.json: no such file or directory"
level=INFO ts=2024-10-30T07:52:20.650098961Z app=security-proxy-auth source=secret.go:79 msg="Reading secret store configuration and authentication token"

Workaround tried:
Restarted the security-proxy-setup, security-secretstore-setup, and security-bootstrapper containers along with this but it did not resolve the issue.

Requesting to provide some help on this issue.

[cloudxxx8]

Restarting the security-proxy-setup, security-secretstore-setup, and security-bootstrapper containers should resolve this issue.
Could you also restart the proxy-auth?
Is there anything wrong from the security-proxy-setup and security-secretstore-setup logs?
Are you able to find /tmp/edgex/secrets/security-proxy-auth/secrets-token.json after restarting them?

[mohammedimrankasab]

@cloudxxx8 I have restarted all of the above-mentioned containers and can't see the mentioned file after restarting either. There are no errors in the security-proxy-setup and security-secretstore-setup logs
Not sure but it looks like somehow the secrets-token.json file is getting deleted.

[cloudxxx8]

are you able to recreate this issue in different machines?

[cloudxxx8]

we are discussing how to improve the token rotation in this ticket
https://github.com/orgs/edgexfoundry/discussions/263