[BUG] when using aad-pod-identity, env-injector fail to pull image from ACR #55
Comments
|
I forgot to mention that i am using aad-pod-identity with my akv2k8s setup. |
|
@rgbusato I'm afraid this is most likely a bug in the env-injector. What happens is that you are using aad-pod-identity (custom credentials), but the webhook is trying to use default credentials to download image ( As I can see it, you have 3 options:
As a side note, the upcoming version of env-injector does not download the image, but use remote inspection of the docker image to find the cmd/entrypoint, but it will still need credentials to access ACR. |
torresdal
changed the title
|
thanks for the detailed response @torresdal. |
|
@torresdal I've tried the second option you've mentioned but doesn't seem to work. Here are the steps I took:
I got the same error I was getting before. I think the reason why this doesn't work is because the env-injector pod is trying to pull the image from within application code ? Therefore the I might just use the controller sync mechanism for now until we fix this. I'm not sure on the effort to fix this but if you have any ideas how to easily fix this I might be able to convince some of the Go devs on our team to help you get that done. |
|
We had some other issue like this what we did is to change the creation of any deployment in our cd pipelines. For now every pod has a cmd spec in itself that way the end injector is not trying to download / inspect the docker image to find the entrypoint. |
|
It should use the imagePullSecrets: ...but I cannot rule out a bug. We have a new version coming out very soon. We’ll make sure to create a test for this scenario. The workaround syggested by @ChaosGramer should work, but this might not be a feasible solution for you. |
|
cool. yea that makes sense and should be easy to implement. |
|
Nothing from the options above worked for me yet. @rgbusato were you able to fix this? |
|
@VpavlovLumedic |
|
Hi, @ChaosGramer. I can't share the information about my pod and deployment, but I will try to reproduce an issue later on demo pod which i can share. Right now i can share this info, which is really similar to @rgbusato case:
The error log from env-injector is:
|
|
Do you have a pull secret configured on your pod? |
|
Yes, I tried that workaround with pullSecrets. (Going to remove it soon, because it's not necessary) Also, I have just added CMD stuff and it got better, but i have a private KeyVault and i see a strange IP inside the pod logs, which is getting denied by firewall of KeyVault. Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Client address is not authorized and caller is not a trusted service.\r\nClient address: 52.148.151.235 So now we can say, that the original bug happens when no CMD defined. @ChaosGramer Do you know anything about this IP? ** Added ** |
|
Ok. The steps below fixed it for me.
B.t.w.
|
|
@VpavlovLumedic I think you have enabled service endpoints on your vnet. Az acr Network-rule |
|
Have you tested the variable substitution you are doing in your args. I’m not so sure whether this will work out |
Yes, it works very well. And the variables are invisible through kubectl exec mypod -- sh -c 'echo $var1' |
|
@VpavlovLumedic @ChaosGramer any chance you could verify if this is still an issue in the latest beta (env-injector chart version 1.1.0-beta.28)? Ref #115 We don't have aad-pod-identity in any of our clusters. If its still an issue (which I hope not) - we will be able to fix this before going official with 1.1.0 |
|
We are getting the same error. Below are the logs from controller. @VpavlovLumedic i tried almost same template and still got same error. Logs are stuck after the daemon error. How did you fix? |
|
Thanks for taking the time to test @pvmraghunandan. I looked through the code once more, and there is an issue indeed. When looking for credentials to use for image registries, it does not take into account using custom auth. I'll patch this asap and get the 1.1.0 version out the door. Thanks. |
@ChaosGramer , I used private endpoint. I think there was some mistake in my configuration (i think that i had no Vnet link between azure privatelink and AKS Vnet), because it works flawlessly now via private IP in a clusters Vnet so akv2k8s env injector resolves keyvault public host to a private ip properly now. |
|
Tested it again. Works fine. deployment.yaml: Pod logs: (modified version of deployment.yaml from this page https://akv2k8s.io/tutorials/env-injection/1-secret/) I should mention that i already have containers pulled: Ok, i added imagePullPolicy: Always to deployment.yaml - still fine From kubectl describe pod: Ok, this looks interesting. I've added imagePullPolicy: Always to my own deployment yaml and redeployed. Env injector pod logs are: But still i get my container running and logs from pod are good without any errors inside. B.t.w. i also have istio installed, but this should not matter here. P.S. Sorry but i have to change variable names and obfuscate other things due to privacy concerns. |
|
@VpavlovLumedic are you sure, your running pod got the secrets injected now? ISTIO: |
@ChaosGramer B.t.w. I had to exclude my key vaults private IP from istio-mesh to get it working. I might reconsider it later. Do you suggest to try to create a ServiceEntry for 10.0.0.1 ? |
|
Rolled back to 1.0.2. Everything is working now. I need to restore my playground cluster to test your updates later. |
torresdal
changed the title
|
Hi everyone. Context: Pods for EnvInjector & Controller (infra-secrets namespace) labeled with Problem: Pods fails to start (to create) and secret injection fails Investigation: Inspection of 'Internal error occurred: failed calling webhook "pods.env-injector.admission.spv.no": an error on the server
("{\"response\":{\"uid\":\"accd6cf2-3afd-4dd9-9be5-ae2daf634266\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"failed
to get auto cmd, error: Get https://dodoreg.azurecr.io/v2/: http: non-successful
response (status=401 body=\\\"{\\\\\\\"errors\\\\\\\":[{\\\\\\\"code\\\\\\\":\\\\\\\"UNAUTHORIZED\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"authentication
required, visit https://aka.ms/acr/authorization for more information.\\\\\\\",\\\\\\\"detail\\\\\\\":null}]}\\\\n\\\")\\ncannot
create client for doker registry\\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageBlob\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:59\\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.GetImageConfig\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:49\\nmain.getContainerCmd\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:36\\nmain.mutateContainers\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:107\\nmain.mutatePodSpec\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:229\\nmain.vaultSecretsMutator\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:138\\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate\\n\\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/mutator.go:25\\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).mutatingAdmissionReview\\n\\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:131\\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).Review\\n\\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:123\\ngithub.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review\\n\\t/go/pkg/mod/github.com/slok/[email protected]/pkg/web")
has prevented the request from succeeding'Logs of [infra-secrets-envinjector-7d67b7bdfd-d29pj] time="2020-10-31T17:47:00Z" level=info msg="found pod to mutate in namespace 'default'"
[infra-secrets-envinjector-7d67b7bdfd-d29pj] time="2020-10-31T17:47:00Z" level=info msg="found container 'garbage-collector' to mutate"
[infra-secrets-envinjector-7d67b7bdfd-d29pj] time="2020-10-31T17:47:00Z" level=info msg="checking for env vars containing '@azurekeyvault' in container garbage-collector"
[infra-secrets-envinjector-7d67b7bdfd-d29pj] time="2020-10-31T17:47:00Z" level=info msg="found env var: cookinginfo-blob-connection-string@azurekeyvault"
[infra-secrets-envinjector-7d67b7bdfd-d29pj] time="2020-10-31T17:47:00Z" level=info msg="found env var: cookinginfo-rabbitmq-connection-strings@azurekeyvault"
[infra-secrets-envinjector-7d67b7bdfd-d29pj] time="2020-10-31T17:47:00Z" level=debug msg="getting container command for container 'garbage-collector'"
[infra-secrets-envinjector-7d67b7bdfd-d29pj] time="2020-10-31T17:47:00Z" level=debug msg="no cmd override in kubernetes for container garbage-collector, checking docker image configuration for entrypoint and cmd for dodoreg.azurecr.io/cookinginfo:1.0.15-2"
[infra-secrets-envinjector-7d67b7bdfd-d29pj] time="2020-10-31T17:47:00Z" level=debug msg="using managed identity for acr credentials"
[infra-secrets-envinjector-7d67b7bdfd-d29pj] time="2020-10-31T17:47:00Z" level=debug msg="image(cookinginfo:1.0.15-2) is not from ACR, skip MSI authentication"
[infra-secrets-envinjector-7d67b7bdfd-d29pj] time="2020-10-31T17:47:00Z" level=info msg="using registry https://dodoreg.azurecr.io"
[infra-secrets-envinjector-7d67b7bdfd-d29pj] 2020-10-31T17:47:00.993496Z info registry.ping url=https://dodoreg.azurecr.io/v2/
[infra-secrets-envinjector-7d67b7bdfd-d29pj] time="2020-10-31T17:47:01Z" level=error msg="failed to mutate pod, error: failed to get auto cmd, error: Get https://dodoreg.azurecr.io/v2/: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required, visit https://aka.ms/acr/authorization for more information.\\\",\\\"detail\\\":null}]}\\n\")\ncannot create client for doker registry\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageBlob\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:59\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.GetImageConfig\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:49\nmain.getContainerCmd\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:36\nmain.mutateContainers\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:107\nmain.mutatePodSpec\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:229\nmain.vaultSecretsMutator\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:138\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/mutator.go:25\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).mutatingAdmissionReview\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:131\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).Review\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:123\ngithub.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/internal/instrumenting/instrumenting.go:42\ngithub.com/slok/kubewebhook/pkg/http.HandlerFor.func1\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/http/handler.go:64\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2802\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1890\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"
[infra-secrets-envinjector-7d67b7bdfd-d29pj] 2020-10-31T17:47:01.024417Z info [ERROR] admission webhook error: failed to get auto cmd, error: Get https://dodoreg.azurecr.io/v2/: http: non-successful response (status=401 body="{\"errors\":[{\"code\":\"UNAUTHORIZED\",\"message\":\"authentication required, visit https://aka.ms/acr/authorization for more information.\",\"detail\":null}]}\n")After inspection of source code, I found this call chain: docker/registry/container/Collect This line indicates that image is not from ACR (but image in application from ACR, actually): azure/credentialprovider/acr/GetAcrCredentials This happens because ACR prefix trimmed from image name in this line: docker/registry/container/Collect As temporal workaround, I decided to clone repository and move indicated line of code after ACR credentials check: docker/registry/container/Collect Logs after creation of new pods (with updated image version): [infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=info msg="found pod to mutate in namespace 'default'"
[infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=info msg="found container 'backend' to mutate"
[infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=info msg="checking for env vars containing '@azurekeyvault' in container backend"
[infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=info msg="found env var: geoservice-redis-connectionstring@azurekeyvault"
[infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=info msg="found env var: geoservice-redis-password@azurekeyvault"
[infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=info msg="found env var: geoservice-yandex-map-api-key@azurekeyvault"
[infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=info msg="found env var: geoservice-google-map-api-key@azurekeyvault"
[infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=debug msg="getting container command for container 'backend'"
[infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=debug msg="no cmd override in kubernetes for container backend, checking docker image configuration for entrypoint and cmd for dodoreg.azurecr.io/geoservice:master-95c4b2e"
[infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=debug msg="using managed identity for acr credentials"
[infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=debug msg="azure: using managed identity extension to retrieve access token"
[infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=debug msg="azure: using User Assigned MSI ID to retrieve access token"
[infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=info msg="using registry https://dodoreg.azurecr.io"
[infra-secrets-envinjector-5c776c66bc-62n6b] 2020-10-31T18:13:45.411925Z info registry.ping url=https://dodoreg.azurecr.io/v2/
[infra-secrets-envinjector-5c776c66bc-62n6b] time="2020-10-31T18:13:45Z" level=error msg="failed to mutate pod, error: failed to get auto cmd, error: Get https://dodoreg.azurecr.io/v2/: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required, visit https://aka.ms/acr/authorization for more information.\\\",\\\"detail\\\":null}]}\\n\")\ncannot create client for doker registry\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageBlob\n\t/go/src/pkg/docker/registry/registry.go:59\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.GetImageConfig\n\t/go/src/pkg/docker/registry/registry.go:49\nmain.getContainerCmd\n\t/go/src/cmd/azure-keyvault-secrets-webhook/registry.go:36\nmain.mutateContainers\n\t/go/src/cmd/azure-keyvault-secrets-webhook/pod.go:107\nmain.mutatePodSpec\n\t/go/src/cmd/azure-keyvault-secrets-webhook/pod.go:229\nmain.vaultSecretsMutator\n\t/go/src/cmd/azure-keyvault-secrets-webhook/main.go:138\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/mutator.go:25\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).mutatingAdmissionReview\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:131\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).Review\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:123\ngithub.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/internal/instrumenting/instrumenting.go:42\ngithub.com/slok/kubewebhook/pkg/http.HandlerFor.func1\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/http/handler.go:64\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2802\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1890\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"
[infra-secrets-envinjector-5c776c66bc-62n6b] 2020-10-31T18:13:45.443185Z info [ERROR] admission webhook error: failed to get auto cmd, error: Get https://dodoreg.azurecr.io/v2/: http: non-successful response (status=401 body="{\"errors\":[{\"code\":\"UNAUTHORIZED\",\"message\":\"authentication required, visit https://aka.ms/acr/authorization for more information.\",\"detail\":null}]}\n")After inspection of source code, I found this call chain: docker/registry/container/getAcrCredentials This line is not captured in logs. It seems, method fails - it can't refresh access token from Management Endpoint and returns error But, error returned from method is not logged and swallowed When I decided to look at NMI logs, I found following: E1030 10:28:36.975337 1 server.go:361] failed to get service principal token for pod:infra-secrets/infra-secrets-envinjector-578549bb89-pcz6v with error code 403, error: failed to get service principal token, error: failed to refresh token, error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"}
I1030 10:28:36.975379 1 server.go:192] status (403) took 69062797 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.0.48.26
I1030 10:40:37.871972 1 server.go:358] exception pod infra-secrets/infra-secrets-envinjector-578549bb89-pcz6v token handling
I1030 10:40:37.871994 1 server.go:301] fetching token for user assigned MSI for resource: https://management.core.windows.net/
E1030 10:40:37.948465 1 server.go:361] failed to get service principal token for pod:infra-secrets/infra-secrets-envinjector-578549bb89-pcz6v with error code 403, error: failed to get service principal token, error: failed to refresh token, error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"}
I1030 10:40:37.948502 1 server.go:192] status (403) took 76596720 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.0.48.26
I1030 10:40:40.004810 1 server.go:358] exception pod infra-secrets/infra-secrets-envinjector-578549bb89-pcz6v token handling
I1030 10:40:40.004828 1 server.go:301] fetching token for user assigned MSI for resource: https://management.core.windows.net/
E1030 10:40:40.077414 1 server.go:361] failed to get service principal token for pod:infra-secrets/infra-secrets-envinjector-578549bb89-pcz6v with error code 403, error: failed to get service principal token, error: failed to refresh token, error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"}
I1030 10:40:40.077455 1 server.go:192] status (403) took 72681143 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.0.48.26
I1030 10:46:38.411145 1 server.go:358] exception pod infra-secrets/infra-secrets-envinjector-578549bb89-pcz6v token handling
I1030 10:46:38.411173 1 server.go:301] fetching token for user assigned MSI for resource: https://management.core.windows.net/
E1030 10:46:38.479281 1 server.go:361] failed to get service principal token for pod:infra-secrets/infra-secrets-envinjector-578549bb89-pcz6v with error code 403, error: failed to get service principal token, error: failed to refresh token, error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"}
I1030 10:46:38.479310 1 server.go:192] status (403) took 68229531 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.0.48.26This fails is because of:
So, I decide to add additional azureidentity and azureidentitybinding (with same selector as described here) for generated AKS identity. Logs after creation of new pods (with updated image version): [infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=info msg="found pod to mutate in namespace 'infra-monitoring'"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=info msg="found container 'backend' to mutate"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=info msg="checking for env vars containing '@azurekeyvault' in container backend"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=info msg="found env var: infra-exporter-kusto-and-azure-infra-monitoring-sp-subscription-id@azurekeyvault"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=info msg="found env var: infra-exporter-kusto-and-azure-infra-monitoring-sp-tenant-id@azurekeyvault"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=info msg="found env var: infra-exporter-kusto-and-azure-infra-monitoring-sp-client-id@azurekeyvault"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=info msg="found env var: infra-exporter-kusto-and-azure-infra-monitoring-sp-client-secret@azurekeyvault"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=debug msg="getting container command for container 'backend'"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=debug msg="no cmd override in kubernetes for container backend, checking docker image configuration for entrypoint and cmd for dodoreg.azurecr.io/infra-exporter-kusto-and-azure:kusto-query-probe-2991d6c"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=debug msg="using managed identity for acr credentials"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=debug msg="azure: using managed identity extension to retrieve access token"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=debug msg="azure: using User Assigned MSI ID to retrieve access token"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=debug msg="discovering auth redirects for: dodoreg.azurecr.io"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=debug msg="exchanging an acr refresh_token"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=debug msg="adding ACR docker config entry for: dodoreg.azurecr.io"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=debug msg="found acr gredentials for dodoreg.azurecr.io"
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=info msg="using registry https://dodoreg.azurecr.io"
[infra-secrets-envinjector-5c776c66bc-t6b4h] 2020-10-31T20:16:58.724867Z info registry.ping url=https://dodoreg.azurecr.io/v2/
[infra-secrets-envinjector-5c776c66bc-t6b4h] 2020-10-31T20:16:58.730127Z info registry.manifest.get url=https://dodoreg.azurecr.io/v2/infra-exporter-kusto-and-azure/manifests/kusto-query-probe-2991d6c repository=infra-exporter-kusto-and-azure reference=kusto-query-probe-2991d6c
[infra-secrets-envinjector-5c776c66bc-t6b4h] time="2020-10-31T20:16:58Z" level=error msg="failed to mutate pod, error: failed to get auto cmd, error: Get https://dodoreg.azurecr.io/v2/infra-exporter-kusto-and-azure/manifests/kusto-query-probe-2991d6c: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required, visit https://aka.ms/acr/authorization for more information.\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Name\\\":\\\"infra-exporter-kusto-and-azure\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")\ncannot download manifest for docker image\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageBlob\n\t/go/src/pkg/docker/registry/registry.go:64\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.GetImageConfig\n\t/go/src/pkg/docker/registry/registry.go:49\nmain.getContainerCmd\n\t/go/src/cmd/azure-keyvault-secrets-webhook/registry.go:36\nmain.mutateContainers\n\t/go/src/cmd/azure-keyvault-secrets-webhook/pod.go:107\nmain.mutatePodSpec\n\t/go/src/cmd/azure-keyvault-secrets-webhook/pod.go:229\nmain.vaultSecretsMutator\n\t/go/src/cmd/azure-keyvault-secrets-webhook/main.go:138\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/mutator.go:25\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).mutatingAdmissionReview\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:131\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).Review\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:123\ngithub.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/internal/instrumenting/instrumenting.go:42\ngithub.com/slok/kubewebhook/pkg/http.HandlerFor.func1\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/http/handler.go:64\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2802\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1890\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"
[infra-secrets-envinjector-5c776c66bc-t6b4h] 2020-10-31T20:16:58.805733Z info [ERROR] admission webhook error: failed to get auto cmd, error: Get https://dodoreg.azurecr.io/v2/infra-exporter-kusto-and-azure/manifests/kusto-query-probe-2991d6c: http: non-successful response (status=401 body="{\"errors\":[{\"code\":\"UNAUTHORIZED\",\"message\":\"authentication required, visit https://aka.ms/acr/authorization for more information.\",\"detail\":[{\"Type\":\"repository\",\"Name\":\"infra-exporter-kusto-and-azure\",\"Action\":\"pull\"}]}]}\n")
[infra-secrets-envinjector-5c776c66bc-t6b4h] cannot download manifest for docker imageIt seems that token for ACR successfully generated: azure/credentialprovider/acr/getACRDockerEntryFromARMToken But code fails in this line. After reading some of Open Issues & PR's, I found interesting one. Problem mentioned in issue indicates, that ACR uses OAuth, but library don't support it. As temporal workaround, I decided to replace original repo with that fork (in go.mod file with replace directive). Logs after creation of new pods (with updated image version): [infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=info msg="found pod to mutate in namespace 'infra-monitoring'"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=info msg="found container 'backend' to mutate"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=info msg="checking for env vars containing '@azurekeyvault' in container backend"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=info msg="found env var: infra-exporter-kusto-and-azure-infra-monitoring-sp-subscription-id@azurekeyvault"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=info msg="found env var: infra-exporter-kusto-and-azure-infra-monitoring-sp-tenant-id@azurekeyvault"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=info msg="found env var: infra-exporter-kusto-and-azure-infra-monitoring-sp-client-id@azurekeyvault"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=info msg="found env var: infra-exporter-kusto-and-azure-infra-monitoring-sp-client-secret@azurekeyvault"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=debug msg="getting container command for container 'backend'"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=debug msg="no cmd override in kubernetes for container backend, checking docker image configuration for entrypoint and cmd for dodoreg.azurecr.io/infra-exporter-kusto-and-azure:kusto-query-probe-2991d6c"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=debug msg="using managed identity for acr credentials"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=debug msg="azure: using managed identity extension to retrieve access token"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=debug msg="azure: using User Assigned MSI ID to retrieve access token"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=debug msg="discovering auth redirects for: dodoreg.azurecr.io"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=debug msg="exchanging an acr refresh_token"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=debug msg="adding ACR docker config entry for: dodoreg.azurecr.io"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=debug msg="found acr gredentials for dodoreg.azurecr.io"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:24Z" level=info msg="using registry https://dodoreg.azurecr.io"
[infra-secrets-envinjector-578549bb89-24cph] 2020-10-31T20:33:24.391065Z info registry.ping url=https://dodoreg.azurecr.io/v2/
[infra-secrets-envinjector-578549bb89-24cph] 2020-10-31T20:33:24.395673Z info registry.manifest.get url=https://dodoreg.azurecr.io/v2/infra-exporter-kusto-and-azure/manifests/kusto-query-probe-2991d6c repository=infra-exporter-kusto-and-azure reference=kusto-query-probe-2991d6c
[infra-secrets-envinjector-578549bb89-24cph] 2020-10-31T20:33:24.839260Z info registry.blob.download url=https://dodoreg.azurecr.io/v2/infra-exporter-kusto-and-azure/blobs/sha256:82f2485c18d8ce7bd42b5d3303eee8b38cde845d6dcc92d4527fbd331a5d13e1 repository=infra-exporter-kusto-and-azure digest=sha256:82f2485c18d8ce7bd42b5d3303eee8b38cde845d6dcc92d4527fbd331a5d13e1
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:25Z" level=info msg="using '/azure-metrics-exporter' as arguments for env-injector"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:25Z" level=debug msg="signed arguments to prevent override"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:25Z" level=debug msg="public signing key for argument verification: \n-----BEGIN RSA PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA62wJm6Ogsx1wVDZQzY3Z\nV7ORiNw13URvb8E7A6rHrtZpUWs3E+90NFTu2W2vbqN4POF07f6tumKeyaqO+HES\n1P9uGmNiOU9ian7SUaEG3QA0uZFtA6W4hHAbiUvLirFk9fVpkWpPworAtEkAC/ou\np7p+uV4NTCtdUjbYhEawYpRxcTKClgyPT980iB+43TJX+FveJVimacUA+bWpaiE2\n6Fgz6TnP+K0siTAaiQnkuZwrYG1SJTlREkTAqh0lIOE4nlQtLxU0uPTYG7G8WRGL\n18XkJAeJfGZkRRERd97KD3gpEI60heVl2uNvUXZ7q6yr3QaoyAzhXlBwX8xxjI+W\ntwIDAQAB\n-----END RSA PUBLIC KEY-----\n"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:25Z" level=debug msg="full exec path: /azure-keyvault/azure-keyvault-env"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:25Z" level=debug msg="container args: [/azure-metrics-exporter]"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:25Z" level=debug msg="mounting volume 'azure-keyvault-env' to '/azure-keyvault/'"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:25Z" level=debug msg="setting ENV_INJECTOR_USE_AUTH_SERVICE=true for container backend"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:25Z" level=info msg="containers mutated and pod updated with init-container and volumes"
[infra-secrets-envinjector-578549bb89-24cph] time="2020-10-31T20:33:30Z" level=info msg="served oauth token to 'infra-monitoring/infra-exporter-kusto-and-azure-fc8b6f96b-j8qlr' at address '10.0.48.47:47090'"Indicates that pods successfully mutated. After that, some errors generated in application pod logs (EnvInjectorEnv init container part): time="2020-10-31T20:33:30Z" level=info msg="using sentralized akv2k8s auth service for authentiction with azure key vault" application=env-injector component=akv2k8s namespace=infra-monitoring
time="2020-10-31T20:33:30Z" level=info msg="found original container command to be /azure-metrics-exporter [/azure-metrics-exporter]" application=env-injector component=akv2k8s namespace=infra-monitoring
time="2020-10-31T20:33:30Z" level=info msg="requesting azure key vault oauth token from https://infra-secrets-envinjector.infra-secrets.svc:443/auth/infra-monitoring/infra-exporter-kusto-and-azure-fc8b6f96b-j8qlr" application=env-injector component=akv2k8s namespace=infra-monitoring
time="2020-10-31T20:33:30Z" level=fatal msg="failed to read secret 'infra-monitoring-sp-client-secret', error keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"Forbidden\" Message=\"The user, group or application 'appid=14080efa-252e-4c8b-b690-e0828bdf4ae8;oid=b7f13d1f-db3c-47c4-9f5f-48c54614ad52;iss=https://sts.windows.net/2b8d2a3c-bc5d-4c9e-9de5-9219a69d6524/' does not have secrets get permission on key vault 'ld-keyvault-dodo;location=westeurope'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287\" InnerError={\"code\":\"AccessDenied\"}" application=env-injector component=akv2k8s namespace=infra-monitoringThis happens because of aad-pod-identity: cached access token token for cluster identity (with name -agentpool) used to access KeyVaults. Conclusions: Long story short, we have following options:
What do you think about discovered issues (trimmed ACR prefix, swallowed error and stale library)? Code can be found in this repo: https://github.com/ArtemTrofimushkin/azure-key-vault-to-kubernetes/tree/fix-aad-pod-identity Also, I can attach our values.yaml. |
|
First of all, thanks for taking the time to dig so deep into this issue @ArtemTrofimushkin and find the solutions you did! I've read through your discoveries, but I need to sit down to create some unit tests and verify the issues you point out and get a cluster going with One of the issues here, I suspect, is that both If you could share your |
|
Sure, this is our values.yaml: controller:
enabled: true
env:
AZURE_ENVIRONMENT: AzurePublicCloud
image:
tag: 1.1.0
keyVault:
customAuth:
enabled: true
logLevel: Trace
crds:
create: true
enabled: true
keep: true
env_injector:
caBundleController:
env:
AZURE_ENVIRONMENT: AzurePublicCloud
image:
tag: 1.1.0
logLevel: Trace
dockerImageInspection:
timeout: '120'
enabled: true
envImage:
tag: 1.1.1
image:
tag: 1.1.10
keyVault:
customAuth:
enabled: true
useAuthService: true
webhook:
certificate:
useCertManager: true
env:
AZURE_ENVIRONMENT: AzurePublicCloud
failurePolicy: Fail
logLevel: Trace
rbac:
create: true
runningInsideAzureAks: trueWe use chart akv2k8s (version 1.1.24) from this commit |
|
Hello, I am observing very similar issue as @ArtemTrofimushkin. I have new AKS cluster using MSI. This cluster has aad-pod-identity installed. Now -- I get the same authentication error as @ArtemTrofimushkin when trying to connect to ACR. Based on my observation, the root problem is that MSI cluster identity is for some reason not used for ACR login. Lumir |
|
Thanks @LumirH - I agree it seams to be related to how we handle ACR. What happens if you set |
|
On my side still the same error:
|
|
Sorry, to come back again but after recreating my cluster I have a problem. Obviously, without injector, everything works OK |
|
@torresdal I would appreciate if you can look at my question I have sent 3 days ago. thank you. |
|
@leonids2005 currently working on a fix for ACR auth. As for your current problem that does seam to be related to something else, and not something I've got reported from others (as far as I can remember). Error says 404 (not found) on the auth endpoint of env-injector at |
|
@torresdal I have installed it using the new chart and have problems - at least with old version pod successfully mutated and with new version, there is again a problem accessing ACR |
|
@leonids2005 -- thats exactly the same issue I'm facing. The workaround is to specify the "command" in the Kubernetes pod specification, so the connection to ACR is not needed. |
|
@LumirH thank you. This is really strange because the version I used before did work. Ok, I will try to use the command. UPDATE: using command works but as I said it did work before without using command. |
|
@torresdal do you have any updates for this item? |
|
Hello, I am facing the issues pointed by @LumirH and @ArtemTrofimushkin . I am using 1.1.28 Helm version (https://charts.spvapi.no) on a new AKS cluster using MSI and secrets are injected correctly when the image is from public repos, however for our private repo in ACR I receive the following: @torresdal do we have any updates on this issue? |
|
So it seams (after talking in the Slack #beta-testing channel) that @LumirH has confirmed that the latest 1.2 beta has solved this issue. Can anyone else verify this? |
|
I can confirm that the latest beta works for me as well 👍 |
|
Have multiple confirmations that this works in latest beta. Will close and hopefully never re-open 😸 |
|
Looks like 1.2 finally fixed this. Also I'm glad to move from 1.0.2 to 1.2 due to security improvements in 1.2 (finally I can set runAsNonRoot: true for entire pod) Thanks for fixes guys! |
Note: Make sure to check out known issues (https://github.com/sparebankenvest/azure-key-vault-to-kubernetes#known-issues) before submitting
Your question
Does the env-injector use some custom mechanism to authenticate with a private docker registry to inspect the image in order to overwrite the entrypoint or cmd?
I'm having some issues where the env-injector container is showing logs where it attempts to pull the image from ACR but it fails.
To Reproduce
If question relates to a certain behavior, describe steps to reproduce:
I'm using AKS + Azure Container Registry with a service principle authentication mechanism that allows me to spin up containers in AKS available in my private registry (ACR).
The original pod itself was deployed successfully and i can verify that the AKS cluster has the necessary permission to pull images from ACR.
However, I've noticed that not env-injector debug logs were showing up in my container so i poked around the env-injector container and noticed some error messages related to it's inability to pull docker images from ACR
Logs
Here is the command I am running to find that information:
$ kubectl logs -n akv2k8s azure-key-vault-env-injector-5458b4bc9d-njxrz -c azure-key-vault-env-injectorHere are the logs indicating the problem mentioned:
The text was updated successfully, but these errors were encountered: