Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stuck on installing airbyte w/ eks + helm #52091

Open
gregmaghakian opened this issue Jan 22, 2025 · 16 comments
Open

Stuck on installing airbyte w/ eks + helm #52091

gregmaghakian opened this issue Jan 22, 2025 · 16 comments
Labels
area/platform issues related to the platform community helm team/deployments type/bug Something isn't working

Comments

@gregmaghakian
Copy link

gregmaghakian commented Jan 22, 2025 •
edited
Loading

Helm Chart Version

1.4.0

What step the error happened?

On deploy

Relevant information

Hi! New to making issues, so hopefully i've provided enough information. please let me know if more is needed.
https://docs.airbyte.com/deploying-airbyte/#5-installing-airbyte i am following the documentation here and am stuck on the helm install section.
I have added created a user in aws that has secrets and s3 perms too. these are the access keys in the secret
I have created a new db that i have added in the values and secret
i have made a namespace airbyte
I have also deployed these secrets and they exist in the airbyte namespace

here is my values.yaml


postgresql:
  enabled: false
global:
  storage:
    type: S3
    secretName: airbyte-config-secrets # Name of your Kubernetes secret.
    bucket: ## S3 bucket names that you've created. We recommend storing the following all in one bucket.
      log: ta-airbyte-bucket 
      state: ta-airbyte-bucket 
      workloadOutput: ta-airbyte-bucket 
    s3:
      region: us-west-2 ## e.g. us-east-1
      authenticationType: instanceProfile ## Use "credentials" or "instanceProfile"
  secretsManager:
      type: awsSecretManager
      secretName: airbyte-config-secrets # Name of your Kubernetes secret.
      awsSecretManager:
        region: us-west-2
        authenticationType: instanceProfile ## Use "credentials" or "instanceProfile"

  database:
    type: external

    # -- Secret name where database credentials are stored
    secretName: airbyte-config-secrets # e.g. "airbyte-config-secrets"

    # -- The database host
    host: airbyte-prod-instance-1.cyk6n2fhagem.us-west-2.rds.amazonaws.com

    # -- The database port
    port: 5432

    # -- The database name
    database: postgres

    # -- The database user
    userSecretKey: "database-user" # e.g. "database-user"

    # -- The key within `secretName` where password is stored
    passwordSecretKey: "database-password" # e.g."database-password"

my secrets: where i remove the values here.

apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
  namespace: airbyte
type: Opaque
stringData:
  # AWS S3 Secrets
  s3-access-key-id: 
  s3-secret-access-key: 
  aws-secret-manager-access-key-id: 
  aws-secret-manager-secret-access-key: 
  database-user: 
  database-password:

Relevant log output

helm install \                     
airbyte \
airbyte/airbyte \
--namespace airbyte \
--values values.yaml --replace --debug               
install.go:225: 2025-01-22 15:17:37.911873 -0500 EST m=+0.030072626 [debug] Original chart version: ""
install.go:242: 2025-01-22 15:17:38.460833 -0500 EST m=+0.579033917 [debug] CHART PATH: /Users/gregmaghakian/Library/Caches/helm/repository/airbyte-1.4.0.tgz

client.go:486: 2025-01-22 15:17:44.045017 -0500 EST m=+6.163224876 [debug] Starting delete for "airbyte-admin" ServiceAccount
wait.go:104: 2025-01-22 15:17:44.182006 -0500 EST m=+6.300214542 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2025-01-22 15:17:45.483432 -0500 EST m=+7.601642292 [debug] creating 1 resource(s)
client.go:486: 2025-01-22 15:17:45.593229 -0500 EST m=+7.711438709 [debug] Starting delete for "airbyte-admin-role" Role
wait.go:104: 2025-01-22 15:17:45.731063 -0500 EST m=+7.849273209 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2025-01-22 15:17:47.029188 -0500 EST m=+9.147400126 [debug] creating 1 resource(s)
client.go:486: 2025-01-22 15:17:47.143066 -0500 EST m=+9.261278084 [debug] Starting delete for "airbyte-admin-binding" RoleBinding
wait.go:104: 2025-01-22 15:17:47.256359 -0500 EST m=+9.374571542 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2025-01-22 15:17:48.212722 -0500 EST m=+10.330935501 [debug] creating 1 resource(s)
client.go:486: 2025-01-22 15:17:48.334057 -0500 EST m=+10.452271167 [debug] Starting delete for "airbyte-airbyte-env" ConfigMap
wait.go:104: 2025-01-22 15:17:48.437185 -0500 EST m=+10.555398751 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
Error: INSTALLATION FAILED: failed pre-install: unable to build kubernetes object for pre-install hook airbyte/templates/env-configmap.yaml: error validating "": error validating data: unknown object type "nil" in ConfigMap.data.JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET
helm.go:86: 2025-01-22 15:17:49.339437 -0500 EST m=+11.457651751 [debug] failed pre-install: unable to build kubernetes object for pre-install hook airbyte/templates/env-configmap.yaml: error validating "": error validating data: unknown object type "nil" in ConfigMap.data.JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET
INSTALLATION FAILED
main.newInstallCmd.func2
	helm.sh/helm/v3/cmd/helm/install.go:158
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/[email protected]/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/[email protected]/command.go:1117
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/[email protected]/command.go:1041
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:85
runtime.main
	runtime/proc.go:272
runtime.goexit
	runtime/asm_arm64.s:1223
@gregmaghakian gregmaghakian added area/platform issues related to the platform needs-triage type/bug Something isn't working labels Jan 22, 2025
@HaJunYoo
Copy link

The issue was present in version 1.4.0. After downgrading to version 1.3.2, the problem was resolved.

airbytehq/airbyte-platform#383

@gregmaghakian
Copy link
Author

gregmaghakian commented Jan 23, 2025 •
edited
Loading

@HaJunYoo on building with 1.3.2

helm install \
airbyte \
airbyte/airbyte \
--namespace airbyte \
--values values.yaml --replace --version 1.3.2 --debug

I am getting

install.go:225: 2025-01-23 09:55:04.566912 -0500 EST m=+0.029903584 [debug] Original chart version: "1.3.2"
install.go:242: 2025-01-23 09:55:05.02228 -0500 EST m=+0.485270876 [debug] CHART PATH: /Users/gregmaghakian/Library/Caches/helm/repository/airbyte-1.3.2.tgz

client.go:486: 2025-01-23 09:55:10.182384 -0500 EST m=+5.645376293 [debug] Starting delete for "airbyte-admin" ServiceAccount
client.go:490: 2025-01-23 09:55:10.30571 -0500 EST m=+5.768702376 [debug] Ignoring delete failure for "airbyte-admin" /v1, Kind=ServiceAccount: serviceaccounts "airbyte-admin" not found
wait.go:104: 2025-01-23 09:55:10.305866 -0500 EST m=+5.768858043 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2025-01-23 09:55:10.946367 -0500 EST m=+6.409359709 [debug] creating 1 resource(s)
client.go:486: 2025-01-23 09:55:11.061328 -0500 EST m=+6.524320626 [debug] Starting delete for "airbyte-admin-role" Role
wait.go:104: 2025-01-23 09:55:11.194471 -0500 EST m=+6.657463251 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2025-01-23 09:55:11.724926 -0500 EST m=+7.187919501 [debug] creating 1 resource(s)
client.go:486: 2025-01-23 09:55:11.8576 -0500 EST m=+7.320593084 [debug] Starting delete for "airbyte-admin-binding" RoleBinding
wait.go:104: 2025-01-23 09:55:11.975476 -0500 EST m=+7.438469418 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2025-01-23 09:55:12.51084 -0500 EST m=+7.973834126 [debug] creating 1 resource(s)
client.go:486: 2025-01-23 09:55:12.6354 -0500 EST m=+8.098393626 [debug] Starting delete for "airbyte-airbyte-env" ConfigMap
client.go:490: 2025-01-23 09:55:12.770465 -0500 EST m=+8.233459251 [debug] Ignoring delete failure for "airbyte-airbyte-env" /v1, Kind=ConfigMap: configmaps "airbyte-airbyte-env" not found
wait.go:104: 2025-01-23 09:55:12.770572 -0500 EST m=+8.233565876 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2025-01-23 09:55:13.345112 -0500 EST m=+8.808106584 [debug] creating 1 resource(s)
client.go:486: 2025-01-23 09:55:13.518134 -0500 EST m=+8.981128626 [debug] Starting delete for "airbyte-airbyte-secrets" Secret
client.go:490: 2025-01-23 09:55:13.637751 -0500 EST m=+9.100745709 [debug] Ignoring delete failure for "airbyte-airbyte-secrets" /v1, Kind=Secret: secrets "airbyte-airbyte-secrets" not found
wait.go:104: 2025-01-23 09:55:13.638021 -0500 EST m=+9.101015834 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
Error: INSTALLATION FAILED: failed pre-install: unable to build kubernetes object for pre-install hook airbyte/templates/secret.yaml: error validating "": error validating data: [unknown object type "nil" in Secret.stringData.AWS_ACCESS_KEY_ID, unknown object type "nil" in Secret.stringData.AWS_SECRET_ACCESS_KEY, unknown object type "nil" in Secret.stringData.DATABASE_PASSWORD]
helm.go:86: 2025-01-23 09:55:14.251325 -0500 EST m=+9.714320709 [debug] failed pre-install: unable to build kubernetes object for pre-install hook airbyte/templates/secret.yaml: error validating "": error validating data: [unknown object type "nil" in Secret.stringData.AWS_ACCESS_KEY_ID, unknown object type "nil" in Secret.stringData.AWS_SECRET_ACCESS_KEY, unknown object type "nil" in Secret.stringData.DATABASE_PASSWORD]
INSTALLATION FAILED
main.newInstallCmd.func2
	helm.sh/helm/v3/cmd/helm/install.go:158
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/[email protected]/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/[email protected]/command.go:1117
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/[email protected]/command.go:1041
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:85
runtime.main
	runtime/proc.go:272
runtime.goexit
	runtime/asm_arm64.s:1223

@gregmaghakian
Copy link
Author 

kubectl describe secrets airbyte-config-secrets -n airbyte
Name:         airbyte-config-secrets
Namespace:    airbyte
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
aws-secret-manager-secret-access-key:  56 bytes
database-password:                     40 bytes
database-user:                         12 bytes
s3-access-key-id:                      28 bytes
s3-secret-access-key:                  56 bytes
aws-secret-manager-access-key-id:      28 bytes

@abuchanan-airbyte
Copy link
Contributor

Can you share your helm version please?

@gregmaghakian
Copy link
Author

gregmaghakian commented Jan 23, 2025 •
edited
Loading 

helm version 
version.BuildInfo{Version:"v3.17.0", GitCommit:"301108edc7ac2a8ba79e4ebf5701b0b6ce6a31e4", GitTreeState:"clean", GoVersion:"go1.23.4"}

this is the output, thank you @abuchanan-airbyte !

do i perhaps need to downgrade?

@abuchanan-airbyte
Copy link
Contributor

What version of k8s are you running?

@abuchanan-airbyte
Copy link
Contributor

I was finally able to reproduce this. It seems specific to a k8s version. The latest version I could reproduce the problem on was v1.24.17 (v1.25+ worked).

So, I'm curious if that aligns with the versions of the people having problems here.

We can be more defensive about missing values in the charts (e.g. airbytehq/airbyte-platform#383) so we'll work on that.

@gregmaghakian
Copy link
Author

Client Version: v1.26.2
Kustomize Version: v4.5.7

@gregmaghakian
Copy link
Author

@abuchanan-airbyte ill try upgrading it. did the values yaml i sent work?

@gregmaghakian
Copy link
Author

sorry also kubernetes version 1.24 🤦 .

@abuchanan-airbyte
Copy link
Contributor

Are you able to upgrade to 1.25+?

@gregmaghakian
Copy link
Author

Yes I am doing that this week after we upgrade some dependencies. After that, it should work with my current set up?

@abuchanan-airbyte
Copy link
Contributor

Yes, hopefully that will resolve your issue

@jamezrin
Copy link

Just for the record, I had this issue before submitting my fix in PR airbytehq/airbyte-platform#383 with the following versions:

$ kubectl version
Client Version: v1.32.1
Kustomize Version: v5.5.0
Server Version: v1.32.0+k3s1

$ helm version
version.BuildInfo{Version:"v3.17.0", GitCommit:"301108edc7ac2a8ba79e4ebf5701b0b6ce6a31e4", GitTreeState:"", GoVersion:"go1.23.5"}

@gregmaghakian
Copy link
Author

hey @abuchanan-airbyte it looks like the secrets arent being picked up properly.
i fixed the other issues. am now on kubernetes 1.25 and running

helm install \
airbyte \
airbyte/airbyte \
--namespace airbyte \
--values values.yaml --replace --version 1.3.2 --debug

i receive a timeout error in the logs with a pwd validation failure for my rds instance.
in looking at the secrets that are made when running the above command, i can see they have 0 bytes whereas the secrets i made and supplied in the values.yaml does have bytes. note: they are encoded

@gregmaghakian
Copy link
Author 

kubectl describe secrets airbyte-airbyte-secrets -n airbyte
Name:         airbyte-airbyte-secrets
Namespace:    airbyte
Labels:       app.kubernetes.io/instance=airbyte
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=airbyte
              app.kubernetes.io/version=1.3.2
              helm.sh/chart=airbyte-1.3.2
Annotations:  helm.sh/hook: pre-install,pre-upgrade
              helm.sh/hook-weight: -1

Type:  Opaque

Data
====
AWS_SECRET_ACCESS_KEY:      0 bytes
DATABASE_PASSWORD:          0 bytes
DATABASE_USER:              8 bytes
WORKLOAD_API_BEARER_TOKEN:  32 bytes
AWS_ACCESS_KEY_ID:          0 bytes

vs.



kubectl describe secrets airbyte-config-secrets -n airbyte 
Name:         airbyte-config-secrets
Namespace:    airbyte
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
s3-access-key-id:                      28 bytes
s3-secret-access-key:                  56 bytes
aws-secret-manager-access-key-id:      28 bytes
aws-secret-manager-secret-access-key:  56 bytes
database-password:                     40 bytes
database-user:                         12 bytes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/platform issues related to the platform community helm team/deployments type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@marcosmarxm @jamezrin @gregmaghakian @HaJunYoo @octavia-squidington-iii @abuchanan-airbyte