[k8s] Template for Slurm-like NFS on Kubernetes

[romilbhardwaj]

Adds a template showcasing how to do slurm-like NFS on Kubernetes. With this template, the specified NFS_USERNAME's home directory will be used as the SkyPilot task's home directory.

Tested with multi-node k3s cluster with EFS acting as the underlying shared storage.

This is purely in user space (i.e., no code changes to SkyPilot, defined purely in the task.yaml). For deeper integration, we can consider adding a field in config that automatically sets this up and passes the current active username to be used for mounting.

Some gotchas:

• We use the image's default username (e.g., sky or root) and don't change it to NFS_USERNAME. Not sure if any applications rely on it.
• Since the home dir is set in user space, there may be some initialization that happens before in ~, notably sky_workdir sync etc. Noted in the README.

Other alternatives we considered:

• Directly mount the NFS at /home/sky. This is problematic because sky logs, sky workdir and skypilot-runtime will get shared across multi-node workers.

[romilbhardwaj]

Another requirement is to write files as a specific user/group to the NFS. By default, in the current implementation they would get written as whatever user/uid is baked in our base image.

In k8s, this can be emulated with:

experimental:
  config_overrides:
    kubernetes:
      pod_config:
        spec:
          # Set security context for proper file ownership
          securityContext:
            runAsUser: 1001
            runAsGroup: 1001
            fsGroup: 1001

However, SkyPilot setup fails with:

D 03-17 15:50:15 subprocess_utils.py:86] Using 1 workers for file mounts.
E 03-17 15:50:15 subprocess_utils.py:151] mkdir: cannot create directory ‘/home/sky/.sky’: Permission denied
E 03-17 15:50:15 subprocess_utils.py:151] command terminated with exit code 1

We can update our base image dir permissions to make this work, but this is not a robust solution, especially when users can come with their own custom images. For user's looking for a workaround, they can build an image and grant all their required uids rwx permissions in the default $HOME directory in the image.

[romilbhardwaj]

Also, we need to fix #4975 before this example can work.

[github-actions]

This PR is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR was closed because it has been stalled for 10 days with no activity.