Skip to content

WIP: deploy flyte-demo (k3s) as a Union app#986

Draft
pingsutw wants to merge 4 commits intomainfrom
oss-binary-app
Draft

WIP: deploy flyte-demo (k3s) as a Union app#986
pingsutw wants to merge 4 commits intomainfrom
oss-binary-app

Conversation

@pingsutw
Copy link
Copy Markdown
Member

Summary

  • Custom Dockerfile.flyte_binary to make ghcr.io/flyteorg/flyte-demo:nightly usable as an app base — k3s rootfs has no /usr/bin, static busybox, and only a partial glibc bundled for Postgres. Installs python-build-standalone, stubs libutil/libnsl/libresolv (merged into libc in glibc 2.34+), sets SSL_CERT_FILE/SSL_CERT_DIR so pyqwest can load CA certs.
  • flyte_binary_app.py wires the image via Image.from_dockerfile(...), sets command=["/bin/k3d-entrypoint.sh", ...], and attaches a PodTemplate requesting privileged=true + SYS_ADMIN/NET_ADMIN.

Status — WIP

  • Image builds and runs locally; flyte imports cleanly.
  • App deploys to Union; the pod reaches the cluster but crashes on startup because k3s needs privileges the pod doesn't have.
  • The PodTemplate securityContext is present in the Union app spec but is silently stripped before reaching the Knative Configuration — Knative generation never advances after the privileged deploy. So the privileged request is a no-op on this tenant.

Test plan

  • Decide whether to pursue privileged app pods on Union (needs platform support) or redesign without k3s-in-a-pod.
  • If privileged path is viable, add the volumeMounts k3s needs (/sys/fs/cgroup, /lib/modules, /var/lib/rancher).
  • File a separate issue for the silent-drop of securityContext in the app control plane.

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Custom Dockerfile installs python-build-standalone, stubs the libs pg-glibc
is missing (libutil/libnsl/libresolv), and points SSL_CERT_FILE at the
bundled CA bundle so pyqwest can talk to the control plane. App env runs
the base k3d entrypoint and requests a privileged pod via PodTemplate.

Note: Union control plane currently drops securityContext before it reaches
Knative, so the privileged request is a no-op. Tracking separately; this
PR is for review of the image/app plumbing only.

Signed-off-by: Kevin Su <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant