Skip to content

Kubeflow unauthenticated instance setup script #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open

Kubeflow unauthenticated instance setup script #116

wants to merge 18 commits into from

Conversation

grandsilva
Copy link
Contributor

according to this PR.

@grandsilva grandsilva mentioned this pull request Feb 24, 2025
Open
@v1ktor0t
Copy link

Hey @grandsilva, thank you for your contribution!

I've been attempting to setup the testbed for some time now, but haven't been able to do so. Mainly, the issue that I'm running into is at the point where the kustomize example is being applied. This is part of the output I'm getting:

Error from server (InternalError): Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.226.84:443: connect: connection refused
Error from server (InternalError): Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.226.84:443: connect: connection refused
Error from server (InternalError): Internal error occurred: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator": could not get REST client: unable to load root certificates: unable to parse bytes as PEM block
Error from server (InternalError): Internal error occurred: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator": could not get REST client: unable to load root certificates: unable to parse bytes as PEM block

I've tried the setup on multiple different machines, with the same result.

Could you please check if you're seeing the same result as me? Also, are you deploying the testbed on a physical, virtual machine, cloud instance?

@grandsilva
Copy link
Contributor Author

grandsilva commented Feb 26, 2025 •
edited
Loading

I saw this error a lot. I only tried to delete the minikube cluster and start a new cluster :)), and that worked. Sadly, I don't know how to fix the issue. I deployed the minikube cluster on my physical Kubuntu desktop. Previously, I ran into a Kubuntu desktop in VirtualBox, too.

@grandsilva
Copy link
Contributor Author

This issue can also occur if multiple network interfaces have multiple default gateways. Could you try to simplify the network environment since Minikube is primarily designed for test environments?

@v1ktor0t
Copy link

hey @grandsilva.

i spent quite a significant amount of time trying to deploy the testbed, but was not successful. i suspect the issue is that the Kubernetes resources are not being properly deployed. here's a sample of the output i'm getting:

Port forward the dex login ...

Not all pods are in Running state in namespace: cert-manager
Not all pods are in Running state in namespace: cert-manager
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
...
No resources found in knative-eventing namespace.
No pods found in namespace: knative-eventing

this might also be the reason why i was never able to obtain a session using the test_dex_login.py script. the service is available on http://localhost:8080, however, after logging in manually i get the following error:

no healthy upstreams

i'm not familiar with Kubeflow to be able to debug the issue further. i tried to deploy it on different host (VMs, cloud instances, etc.) multiple times, with basically the same result.

i'd suggest simplifying the deployment to only the necessary components, which should make deployment more stable and consistent.

on an additional note: you can assume that Docker and Kubernetes clusters will already be deployed on the machine. so there is no reason to install them yourself. furthermore, i'd suggest refraining from installing any dependencies system-side. for example:

tar -xzvf kustomize_v5.4.3_linux_amd64.tar.gz
chmod a+x kustomize
sudo mv kustomize /usr/local/bin

those can be safely run from the current working directory. this will make sure the system doesn't get polluted with different versions of application binaries.

@grandsilva
Copy link
Contributor Author

@v1ktor0t I'm so sorry for my mistake on this, I'm sure I fixed the bug now, please run the script again.

the reason that we don't get results is that the knative-eventing namespace doesn't exist at all and we get errors, instead of stopping the script we continue to run it.

After I fixed the errors I'm sure the script works fine and it waits until all pods in all related namespaces get ready before we do anything further.

@grandsilva
Copy link
Contributor Author

Also, according to the newly updated README.md, please make sure that you delete the minikube cluster for a new run, this is mostly because I think there maybe some errors if we don't delete the old clusters.

@v1ktor0t
Copy link

v1ktor0t commented Mar 5, 2025

Hey @grandsilva

The deployment script still does not work, even with the changes. In it's current form, the script will ultimately fail with the following error message:

Failed to setup k8s pods

Multiple retries result with the same error message.

Like I said in my initial message, I'd try simplifying the script and the deployed components, until it is in a state where it can be reliably deployed.

@grandsilva
Copy link
Contributor Author

grandsilva commented Mar 11, 2025 •
edited
Loading

@v1ktor0t Hii
I'm terribly sorry for this, the error message is a bug, I always ignore these parts of errors but I'm really confused as to why the script worked for me So because of this, I created a base Ubuntu Server 24.04.2 LTS, and every time I wanted to test the start script, I tried it on a new base instance. now I'm sure the script is working.
I tried to reduce the time of deployments, since I'm not a k8s expert I can create a kubeflow ecosystem with k8s in a reasonable time sadly! with home internet(200-300mbs) we can expect 10 minutes to fully set up the kubeflow.

Also, I noticed according to the official kubeflow manifests repo, we need to use docker login inside the minikube, after logging in we can have faster docker pulls, I see some improvement with this.

@alessandro-Doyensec
Copy link

Hey @grandsilva I'm trying to deploy the testbed, this is the VM I'm using:

user@machine:~/security-testbeds$ neofetch

            .-/+oossssoo+/-.               user@machine
        `:+ssssssssssssssssss+:`           ----------
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 24.04.1 LTS x86_64
    .ossssssssssssssssssdMMMNysssso.       Host: KVM/QEMU (Standard PC (i440FX + PIIX, 1996) pc-i440fx-9.0)
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 6.8.0-57-generic
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 4 mins
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 653 (dpkg)
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 5.2.21
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Resolution: 1280x800
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   Terminal: /dev/pts/0
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: QEMU Virtual version 2.5+ (2) @ 3.699GHz
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   GPU: 00:02.0 Vendor 1234 Device 1111
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Memory: 1236MiB / 15993MiB
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
  +sssssssssdmydMMMMMMMMddddyssssssss+
   /ssssssssssshdmNNNNmyNMMMMhssssss/
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.
The `start.sh` script is not working. This is the output I get from launching your script (it's not the first time I launch it) 
user@machine:~/security-testbeds/kubeflow/exposed_UI$ bash start.sh

This is a Ubuntu Distro


Increase inotify limits to prevent CrashLoopBackOff ...

[sudo] password for user:
fs.inotify.max_user_instances = 1280
fs.inotify.max_user_watches = 655360

Install python3, python3-pip, python3-venv, wget and git ...

Get:1 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Hit:2 http://it.archive.ubuntu.com/ubuntu noble InRelease
Hit:3 https://download.docker.com/linux/ubuntu noble InRelease
Hit:4 http://it.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:5 http://it.archive.ubuntu.com/ubuntu noble-backports InRelease
Get:6 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [741 kB]
Get:7 http://security.ubuntu.com/ubuntu noble-security/main Translation-en [142 kB]
Get:8 http://security.ubuntu.com/ubuntu noble-security/restricted Translation-en [175 kB]
Get:9 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages [829 kB]
Fetched 2012 kB in 1s (1857 kB/s)
Reading package lists... Done
/usr/bin/git
/usr/bin/python3
/usr/bin/pip
usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear] [--upgrade] [--without-pip] [--prompt PROMPT] [--upgrade-deps]
            ENV_DIR [ENV_DIR ...]
venv: error: the following arguments are required: ENV_DIR
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
python3-venv is already the newest version (3.12.3-0ubuntu2).
0 upgraded, 0 newly installed, 0 to remove and 106 not upgraded.

Install Docker ...

/usr/bin/docker

Install minikube ...

./minikube
We need to clean minikube clusters before each run, please enter y if you agree. (y/n): y

Deleting all Minikube clusters...

🔥  Deleting "minikube" in docker ...
🔥  Removing /home/user/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.
🔥  Successfully deleted all profiles
🔥  Successfully deleted all profiles
😄  minikube v1.34.0 on Ubuntu 24.04 (kvm/amd64)
✨  Automatically selected the docker driver. Other choices: none, ssh
📌  Using Docker driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.45 ...
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
🐳  Preparing Kubernetes v1.31.0 on Docker 27.2.0 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Install kubectl...

./kubectl

Install Kustomize ...

./kustomize

Apply manifests for pipelines...

namespace/auth serverside-applied
namespace/cert-manager serverside-applied
namespace/istio-system serverside-applied
namespace/knative-serving serverside-applied
namespace/kubeflow serverside-applied
namespace/oauth2-proxy serverside-applied
customresourcedefinition.apiextensions.k8s.io/authcodes.dex.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/authorizationpolicies.security.istio.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev serverside-applied
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/clusterdomainclaims.networking.internal.knative.dev serverside-applied
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/compositecontrollers.metacontroller.k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev serverside-applied
customresourcedefinition.apiextensions.k8s.io/controllerrevisions.metacontroller.k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/decoratorcontrollers.metacontroller.k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/destinationrules.networking.istio.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/domainmappings.serving.knative.dev serverside-applied
customresourcedefinition.apiextensions.k8s.io/envoyfilters.networking.istio.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/gateways.networking.istio.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev serverside-applied
customresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev serverside-applied
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev serverside-applied
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/peerauthentications.security.istio.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev serverside-applied
customresourcedefinition.apiextensions.k8s.io/profiles.kubeflow.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/proxyconfigs.networking.istio.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/pvcviewers.kubeflow.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/requestauthentications.security.istio.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev serverside-applied
customresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev serverside-applied
customresourcedefinition.apiextensions.k8s.io/scheduledworkflows.kubeflow.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev serverside-applied
customresourcedefinition.apiextensions.k8s.io/serviceentries.networking.istio.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev serverside-applied
customresourcedefinition.apiextensions.k8s.io/sidecars.networking.istio.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/telemetries.telemetry.istio.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/viewers.kubeflow.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/virtualservices.networking.istio.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/wasmplugins.extensions.istio.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/workflowartifactgctasks.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/workflowtaskresults.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/workflowtasksets.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/workloadentries.networking.istio.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/workloadgroups.networking.istio.io serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/cache-webhook-kubeflow serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/pvcviewer-mutating-webhook-configuration serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.istio.networking.internal.knative.dev serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.serving.knative.dev serverside-applied
serviceaccount/dex serverside-applied
serviceaccount/cert-manager serverside-applied
serviceaccount/cert-manager-cainjector serverside-applied
serviceaccount/cert-manager-webhook serverside-applied
serviceaccount/cluster-local-gateway-service-account serverside-applied
serviceaccount/istio-ingressgateway-service-account serverside-applied
serviceaccount/istio-reader-service-account serverside-applied
serviceaccount/istiod serverside-applied
serviceaccount/activator serverside-applied
serviceaccount/controller serverside-applied
serviceaccount/argo serverside-applied
serviceaccount/centraldashboard serverside-applied
serviceaccount/kubeflow-pipelines-cache serverside-applied
serviceaccount/kubeflow-pipelines-container-builder serverside-applied
serviceaccount/kubeflow-pipelines-metadata-writer serverside-applied
serviceaccount/kubeflow-pipelines-viewer serverside-applied
serviceaccount/meta-controller-service serverside-applied
serviceaccount/metadata-grpc-server serverside-applied
serviceaccount/ml-pipeline serverside-applied
serviceaccount/ml-pipeline-persistenceagent serverside-applied
serviceaccount/ml-pipeline-scheduledworkflow serverside-applied
serviceaccount/ml-pipeline-ui serverside-applied
serviceaccount/ml-pipeline-viewer-crd-service-account serverside-applied
serviceaccount/ml-pipeline-visualizationserver serverside-applied
serviceaccount/mysql serverside-applied
serviceaccount/pipeline-runner serverside-applied
serviceaccount/profiles-controller-service-account serverside-applied
^Bserviceaccount/pvcviewer-controller-manager serverside-applied
serviceaccount/oauth2-proxy serverside-applied
networkpolicy.networking.k8s.io/cache-server serverside-applied
networkpolicy.networking.k8s.io/centraldashboard serverside-applied
networkpolicy.networking.k8s.io/default-allow-same-namespace serverside-applied
networkpolicy.networking.k8s.io/metadata-grpc-server serverside-applied
networkpolicy.networking.k8s.io/metatada-envoy serverside-applied
networkpolicy.networking.k8s.io/minio serverside-applied
networkpolicy.networking.k8s.io/ml-pipeline serverside-applied
networkpolicy.networking.k8s.io/ml-pipeline-ui serverside-applied
networkpolicy.networking.k8s.io/model-registry serverside-applied
networkpolicy.networking.k8s.io/poddefaults serverside-applied
networkpolicy.networking.k8s.io/pvcviewer-webhook serverside-applied
networkpolicy.networking.k8s.io/seldon serverside-applied
networkpolicy.networking.k8s.io/spark-operator-webhook serverside-applied
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving serverside-applied
role.rbac.authorization.k8s.io/cluster-local-gateway-sds serverside-applied
role.rbac.authorization.k8s.io/istio-ingressgateway-sds serverside-applied
role.rbac.authorization.k8s.io/istiod serverside-applied
role.rbac.authorization.k8s.io/knative-serving-activator serverside-applied
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection serverside-applied
role.rbac.authorization.k8s.io/cert-manager:leaderelection serverside-applied
role.rbac.authorization.k8s.io/argo-role serverside-applied
role.rbac.authorization.k8s.io/centraldashboard serverside-applied
role.rbac.authorization.k8s.io/kubeflow-pipelines-cache-role serverside-applied
role.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-role serverside-applied
role.rbac.authorization.k8s.io/ml-pipeline serverside-applied
drole.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-role serverside-applied
role.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-role serverside-applied
role.rbac.authorization.k8s.io/ml-pipeline-ui serverside-applied
role.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role serverside-applied
role.rbac.authorization.k8s.io/pipeline-runner serverside-applied
role.rbac.authorization.k8s.io/profiles-leader-election-role serverside-applied
role.rbac.authorization.k8s.io/pvcviewer-leader-election-role serverside-applied
clusterrole.rbac.authorization.k8s.io/aggregate-to-kubeflow-pipelines-edit serverside-applied
clusterrole.rbac.authorization.k8s.io/aggregate-to-kubeflow-pipelines-view serverside-applied
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin serverside-applied
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit serverside-applied
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view serverside-applied
clusterrole.rbac.authorization.k8s.io/argo-cluster-role serverside-applied
clusterrole.rbac.authorization.k8s.io/centraldashboard serverside-applied
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector serverside-applied
clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view serverside-applied
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io serverside-applied
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates serverside-applied
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests serverside-applied
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges serverside-applied
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers serverside-applied
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim serverside-applied
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers serverside-applied
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders serverside-applied
clusterrole.rbac.authorization.k8s.io/cert-manager-edit serverside-applied
clusterrole.rbac.authorization.k8s.io/cert-manager-view serverside-applied
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews serverside-applied
clusterrole.rbac.authorization.k8s.io/dex serverside-applied
clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-istio-system serverside-applied
clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-istio-system serverside-applied
clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-istio-system serverside-applied
clusterrole.rbac.authorization.k8s.io/knative-serving-activator-cluster serverside-applied
clusterrole.rbac.authorization.k8s.io/knative-serving-addressable-resolver serverside-applied
clusterrole.rbac.authorization.k8s.io/knative-serving-admin serverside-applied
clusterrole.rbac.authorization.k8s.io/knative-serving-aggregated-addressable-resolver serverside-applied
clusterrole.rbac.authorization.k8s.io/knative-serving-core serverside-applied
clusterrole.rbac.authorization.k8s.io/knative-serving-istio serverside-applied
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-admin serverside-applied
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-edit serverside-applied
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-view serverside-applied
clusterrole.rbac.authorization.k8s.io/knative-serving-podspecable-binding serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-admin serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-edit serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-istio-admin serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-istio-edit serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-istio-view serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-admin serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-edit serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-view serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-cache-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-edit serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-view serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-view serverside-applied
clusterrole.rbac.authorization.k8s.io/ml-pipeline serverside-applied
clusterrole.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-role serverside-applied
clusterrole.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-role serverside-applied
clusterrole.rbac.authorization.k8s.io/ml-pipeline-ui serverside-applied
clusterrole.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role serverside-applied
clusterrole.rbac.authorization.k8s.io/pvcviewer-metrics-reader serverside-applied
clusterrole.rbac.authorization.k8s.io/pvcviewer-proxy-role serverside-applied
clusterrole.rbac.authorization.k8s.io/pvcviewer-role serverside-applied
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving serverside-applied
rolebinding.rbac.authorization.k8s.io/cluster-local-gateway-sds serverside-applied
rolebinding.rbac.authorization.k8s.io/istio-ingressgateway-sds serverside-applied
rolebinding.rbac.authorization.k8s.io/istiod serverside-applied
rolebinding.rbac.authorization.k8s.io/knative-serving-activator serverside-applied
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection serverside-applied
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection serverside-applied
rolebinding.rbac.authorization.k8s.io/argo-binding serverside-applied
rolebinding.rbac.authorization.k8s.io/centraldashboard serverside-applied
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-binding serverside-applied
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-binding serverside-applied
rolebinding.rbac.authorization.k8s.io/ml-pipeline serverside-applied
rolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-binding serverside-applied
rolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-binding serverside-applied
rolebinding.rbac.authorization.k8s.io/ml-pipeline-ui serverside-applied
rolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-binding serverside-applied
rolebinding.rbac.authorization.k8s.io/pipeline-runner-binding serverside-applied
rolebinding.rbac.authorization.k8s.io/profiles-leader-election-rolebinding serverside-applied
rolebinding.rbac.authorization.k8s.io/pvcviewer-leader-election-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/argo-binding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/centraldashboard serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/dex serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-istio-system serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-istio-system serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-istio-system serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/knative-serving-activator-cluster serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/knative-serving-controller-addressable-resolver serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/knative-serving-controller-admin serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-binding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-binding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/meta-controller-cluster-role-binding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-binding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-binding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-ui serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-binding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/profiles-cluster-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/pvcviewer-manager-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/pvcviewer-proxy-rolebinding serverside-applied
configmap/dex serverside-applied
configmap/istio serverside-applied
configmap/istio-sidecar-injector serverside-applied
configmap/config-autoscaler serverside-applied
configmap/config-defaults serverside-applied
configmap/config-deployment serverside-applied
configmap/config-domain serverside-applied
configmap/config-features serverside-applied
configmap/config-gc serverside-applied
configmap/config-istio serverside-applied
configmap/config-leader-election serverside-applied
configmap/config-logging serverside-applied
configmap/config-network serverside-applied
configmap/config-observability serverside-applied
configmap/config-tracing serverside-applied
configmap/centraldashboard-config serverside-applied
configmap/centraldashboard-parameters serverside-applied
configmap/kfp-launcher serverside-applied
configmap/kubeflow-pipelines-profile-controller-code-hdk828hd6c serverside-applied
configmap/kubeflow-pipelines-profile-controller-env-5252m69c4c serverside-applied
configmap/metadata-grpc-configmap serverside-applied
configmap/ml-pipeline-ui-configmap serverside-applied
configmap/namespace-labels-data-dh6f5dm5ht serverside-applied
configmap/pipeline-api-server-config-dc9hkg52h6 serverside-applied
configmap/pipeline-install-config serverside-applied
configmap/profiles-config-5h9m86f79f serverside-applied
configmap/workflow-controller-configmap serverside-applied
configmap/oauth2-proxy-hk55gm96k4 serverside-applied
configmap/oauth2-proxy-parameters-74659b6648 serverside-applied
configmap/oauth2-proxy-theme-5t624ft8b8 serverside-applied
configmap/default-install-config-9h2h2b6hbk serverside-applied
secret/dex-oidc-client serverside-applied
secret/dex-passwords serverside-applied
secret/routing-serving-certs serverside-applied
secret/knative-serving-certs serverside-applied
secret/net-istio-webhook-certs serverside-applied
secret/routing-serving-certs serverside-applied
secret/serving-certs-ctrl-ca serverside-applied
secret/webhook-certs serverside-applied
secret/mlpipeline-minio-artifact serverside-applied
secret/mysql-secret serverside-applied
secret/oauth2-proxy-h675gf55ht serverside-applied
service/dex serverside-applied
service/cert-manager serverside-applied
service/cert-manager-webhook serverside-applied
service/cluster-local-gateway serverside-applied
service/istio-ingressgateway serverside-applied
service/istiod serverside-applied
service/knative-local-gateway serverside-applied
service/activator-service serverside-applied
service/autoscaler serverside-applied
service/controller serverside-applied
service/net-istio-webhook serverside-applied
service/webhook serverside-applied
service/cache-server serverside-applied
service/centraldashboard serverside-applied
service/kubeflow-pipelines-profile-controller serverside-applied
service/metadata-envoy-service serverside-applied
service/metadata-grpc-service serverside-applied
service/minio-service serverside-applied
service/ml-pipeline serverside-applied
service/ml-pipeline-ui serverside-applied
service/ml-pipeline-visualizationserver serverside-applied
service/mysql serverside-applied
service/profiles-kfam serverside-applied
service/pvcviewer-controller-manager-metrics-service serverside-applied
service/pvcviewer-webhook-service serverside-applied
service/oauth2-proxy serverside-applied
priorityclass.scheduling.k8s.io/workflow-controller serverside-applied
persistentvolumeclaim/minio-pvc serverside-applied
persistentvolumeclaim/mysql-pv-claim serverside-applied
deployment.apps/dex serverside-applied
deployment.apps/cert-manager serverside-applied
deployment.apps/cert-manager-cainjector serverside-applied
deployment.apps/cert-manager-webhook serverside-applied
deployment.apps/cluster-local-gateway serverside-applied
deployment.apps/istio-ingressgateway serverside-applied
deployment.apps/istiod serverside-applied
deployment.apps/activator serverside-applied
deployment.apps/autoscaler serverside-applied
deployment.apps/controller serverside-applied
deployment.apps/net-istio-controller serverside-applied
deployment.apps/net-istio-webhook serverside-applied
deployment.apps/webhook serverside-applied
deployment.apps/cache-server serverside-applied
deployment.apps/centraldashboard serverside-applied
deployment.apps/kubeflow-pipelines-profile-controller serverside-applied
deployment.apps/metadata-envoy-deployment serverside-applied
deployment.apps/metadata-grpc-deployment serverside-applied
deployment.apps/metadata-writer serverside-applied
deployment.apps/minio serverside-applied
deployment.apps/ml-pipeline serverside-applied
deployment.apps/ml-pipeline-persistenceagent serverside-applied
deployment.apps/ml-pipeline-scheduledworkflow serverside-applied
deployment.apps/ml-pipeline-ui serverside-applied
deployment.apps/ml-pipeline-viewer-crd serverside-applied
deployment.apps/ml-pipeline-visualizationserver serverside-applied
deployment.apps/mysql serverside-applied
deployment.apps/profiles-deployment serverside-applied
deployment.apps/pvcviewer-controller-manager serverside-applied
deployment.apps/workflow-controller serverside-applied
deployment.apps/oauth2-proxy serverside-applied
statefulset.apps/metacontroller serverside-applied
poddisruptionbudget.policy/activator-pdb serverside-applied
poddisruptionbudget.policy/webhook-pdb serverside-applied
horizontalpodautoscaler.autoscaling/cluster-local-gateway serverside-applied
horizontalpodautoscaler.autoscaling/istio-ingressgateway serverside-applied
horizontalpodautoscaler.autoscaling/istiod serverside-applied
horizontalpodautoscaler.autoscaling/activator serverside-applied
horizontalpodautoscaler.autoscaling/webhook serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.istio.networking.internal.knative.dev serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.serving.knative.dev serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-istio-system serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/pvcviewer-validating-webhook-configuration serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.serving.knative.dev serverside-applied
resource mapping not found for name: "queue-proxy" namespace: "knative-serving" from "exampleManifest.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"
ensure CRDs are installed first
resource mapping not found for name: "kfp-cache-cert" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "Certificate" in version "cert-manager.io/v1"
ensure CRDs are installed first
resource mapping not found for name: "pvcviewer-serving-cert" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "Certificate" in version "cert-manager.io/v1"
ensure CRDs are installed first
resource mapping not found for name: "kubeflow-self-signing-issuer" namespace: "" from "exampleManifest.yaml": no matches for kind "ClusterIssuer" in version "cert-manager.io/v1"
ensure CRDs are installed first
resource mapping not found for name: "kfp-cache-selfsigned-issuer" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "Issuer" in version "cert-manager.io/v1"
ensure CRDs are installed first
resource mapping not found for name: "pvcviewer-selfsigned-issuer" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "Issuer" in version "cert-manager.io/v1"
ensure CRDs are installed first
resource mapping not found for name: "kubeflow-user-example-com" namespace: "" from "exampleManifest.yaml": no matches for kind "Profile" in version "kubeflow.org/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "kubeflow-pipelines-profile-controller" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "CompositeController" in version "metacontroller.k8s.io/v1alpha1"
ensure CRDs are installed first
resource mapping not found for name: "knative" namespace: "knative-serving" from "exampleManifest.yaml": no matches for kind "DestinationRule" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "metadata-grpc-service" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "DestinationRule" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "ml-pipeline" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "DestinationRule" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "ml-pipeline-minio" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "DestinationRule" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "ml-pipeline-mysql" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "DestinationRule" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "ml-pipeline-ui" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "DestinationRule" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "ml-pipeline-visualizationserver" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "DestinationRule" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "cluster-local-gateway" namespace: "istio-system" from "exampleManifest.yaml": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "istio-ingressgateway" namespace: "istio-system" from "exampleManifest.yaml": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "kubeflow-gateway" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "dex" namespace: "auth" from "exampleManifest.yaml": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "centraldashboard" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "metadata-grpc" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "ml-pipeline-ui" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "profiles-kfam" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "oauth2-proxy" namespace: "oauth2-proxy" from "exampleManifest.yaml": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
ensure CRDs are installed first
resource mapping not found for name: "knative-local-gateway" namespace: "knative-serving" from "exampleManifest.yaml": no matches for kind "Gateway" in version "networking.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "cluster-local-gateway" namespace: "istio-system" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "global-deny-all" namespace: "istio-system" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "istio-ingressgateway" namespace: "istio-system" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "istio-ingressgateway-oauth2-proxy" namespace: "istio-system" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "istio-ingressgateway-require-jwt" namespace: "istio-system" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "activator-service" namespace: "knative-serving" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "autoscaler" namespace: "knative-serving" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "controller" namespace: "knative-serving" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "istio-webhook" namespace: "knative-serving" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "webhook" namespace: "knative-serving" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "central-dashboard" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "metadata-grpc-service" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "minio-service" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "ml-pipeline" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "ml-pipeline-ui" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "ml-pipeline-visualizationserver" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "mysql" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "profiles-kfam" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "service-cache-server" namespace: "kubeflow" from "exampleManifest.yaml": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "net-istio-webhook" namespace: "knative-serving" from "exampleManifest.yaml": no matches for kind "PeerAuthentication" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "webhook" namespace: "knative-serving" from "exampleManifest.yaml": no matches for kind "PeerAuthentication" in version "security.istio.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "dex-jwt" namespace: "istio-system" from "exampleManifest.yaml": no matches for kind "RequestAuthentication" in version "security.istio.io/v1beta1"
ensure CRDs are installed first


Wait until all pods get ready ...

Not all pods are in Running state in namespace: cert-manager
Not all pods are in Running state in namespace: cert-manager
Not all pods are in Running state in namespace: cert-manager
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: knative-serving
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
Not all pods are in Running state in namespace: istio-system
...

The script has been running for a few hours now, can I ask you why 48,131 lines of kubeflow/exposed_UI/exampleManifest.yaml are needed? If you got those from https://github.com/kubeflow/kubeflow/tree/master/components, are they all necessary for the purpose of this testbed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@grandsilva @v1ktor0t @alessandro-Doyensec @am0o0 @tooryx