Unable to properly authenticate with --kubeconfig

[kamilatwork]

I was successfully using kargo login --kubeconfig <host> to log in to our kargo instance. Since updating to 1.9 (tested against 1.9.2) authentication does not seem to work any more:

$ kargo refresh warehouse --project <project> <warehouse>
Error: refresh Warehouse: [POST /v1beta1/projects/{project}/warehouses/{warehouse}/refresh] RefreshWarehouse (status 401): {}

$ kargo version
Client Version: v1.9.2
Error: get version info from server: [GET /v1beta1/system/server-version] GetVersionInfo (status 401): {}

Has anything change? Didn't find anything useful in release notes.

Thanks!

[krancour]

A lot changed in 1.9.

It's not obvious to me that either of these is actually the culprit, but my top two suspicions would be that this bug is an unforeseen consequence of the transition to the REST API or the remediation of this CVE: GHSA-w5wv-wvrp-v5m5

What version were you coming from? If it had to do with remediation of the CVE, 1.6.3, 1.7.7, and 1.8.7 also would have been affected. If you tell me you upgraded from one of those, it narrows down the possible causes.

[kamilatwork]

Thanks for coming back to me. I believe we have started seeing the issue around DOA release of 1.8.8, which we reverted quickly back to 1.8.6 on 2026-01-28, and then updated to 1.8.9 day after (seems like we have skipped 1.8.7).

And yes, everything else, except for Kargo itself, is exactly the same, same AWS EKS cluster.

[krancour]

Notes to self:

The initial statement:

Since updating to 1.9 (tested against 1.9.2) authentication does not seem to work any more

may appear to contradict the later statement:

I believe we have started seeing the issue around DOA release of 1.8.8

However, v1.8.8 was DOA precisely because it was accidentally equivalent to a v1.9.0 RC, so these two statements are actually quite consistent.

As of v1.9.0, the CLI utilizes the new REST API, so I strongly suspect the issue lies in the new API's authentication middleware or the new API's generated client.

[kamilatwork]

You are right, apologies for inconsistency there, it was only after your question that I looked at our history.

I did see in some other issue that '--kubeconfig is rarely used', but we rely on it to refresh warehouses after pushing new images to registries; it worked quite nicely with relatively high interval.

--sso options works as expected, but it's not suitable for CI usage.

[krancour]

apologies for inconsistency there

No apologies necessary. Like I said... those two statements are actually more consistent than they may initially appear.

I did see in some other issue that '--kubeconfig is rarely used'

I do believe this to be the case, and frequency of use aside, it doesn't (and never did) work for any cluster that doesn't authenticate clients using bearer tokens. For instance, it does not work for clusters authenticating clients via mTLS.

--sso options works as expected, but it's not suitable for CI usage

Precisely. But now that you've reminded me of this impetus for using --kubeconfig, it's got me wondering whether the support for API tokens introduced in v1.9.0 might actually have finally rendered that flag obsolete. I'm going to have a discussion about that possibility with the team.

Would the new API token feature satisfy your needs?

Project-level API tokens: https://docs.kargo.io/user-guide/security/api-tokens

More broadly scoped, system-level API tokens: https://docs.kargo.io/operator-guide/security/api-tokens

[kamilatwork]

Would the new API token feature satisfy your needs?

Project-level API tokens: docs.kargo.io/user-guide/security/api-tokens

More broadly scoped, system-level API tokens: docs.kargo.io/operator-guide/security/api-tokens

In theory it would, yes. It would require creating another secret on our CI, which isn't a problem but something that didn't need doing with --kubeconfig since it was already authenticated.

With that aside, I did not manage to create a token (while being authenticated using --sso to admin mapped user:

$ kargo create token --system --role kargo-admin kargo-admin-token-1
Error: create API token: [POST /v1beta1/system/roles/{role}/api-tokens] CreateSystemAPIToken (status 404): {}

Might be related: when I navigate to /settings/access in UI, neither Roles nor API Tokens have any items listed and Role.rbac.authorization.k8s.io "kargo-project-admin" not found error is displayed.

We have Kargo (1.9.2) installed using official helm chart (with adminAccount.enabled: false and oidc set up).