Expand PodOverrides with volumes, volumeMounts, and securityContext#1041
Conversation
Adds four optional fields to the shared PodOverrides struct so that spawned Task pods can attach extra volumes, mount them into the agent container, and declare pod- and container-level securityContext. This unblocks PSS-restricted namespaces and use cases that need extra mounts (writable profile dirs, CA bundles, custom CLI configs). Field semantics: - Volumes: appended to the Job's PodSpec.Volumes; user-supplied names must not collide with the Kelos-managed names "workspace" and "kelos-plugin", and duplicates are rejected. - VolumeMounts: appended to the agent container's mounts. A flat list rather than a map keyed by container name -- only the agent container is exposed; init containers stay private. - PodSecurityContext: replaces the pod-level securityContext set by the controller. As a carve-out, FSGroup is preserved from Kelos's default when the user does not set it explicitly, so the agent user retains read/write access to the workspace volume. - ContainerSecurityContext: replaces the agent container's securityContext (allowPrivilegeEscalation=false, capabilities.drop=[ALL], readOnlyRootFilesystem=true, etc.) so the pod can land in a PSS-restricted namespace. Also fixes a latent bug in helmchart.Render: rendered templates were being split into YAML documents with strings.Split(content, "---\n"), which corrupts documents whose description text happens to contain "---\n". The new corev1 types pulled into the CRD surface this -- PodSecurityContext.fsGroup's description contains the literal example "rw-rw----\n", which the naive split treats as a document boundary. Replaced with k8s.io/apimachinery/pkg/util/yaml's reader, which only splits on lines that exactly match "---". Existing render tests that relied on a substring search for ":latest" were also tightened to match real image references rather than CRD description text that mentions ":latest" in prose.
|
/kelos api-review |
|
/kelos review |
There was a problem hiding this comment.
🤖 Kelos API Reviewer Agent @gjkim42
API Design Review
Verdict: APPROVE
Scope: Adds four optional fields to PodOverrides (volumes, volumeMounts, podSecurityContext, containerSecurityContext), enabling spawned Task pods to attach user-supplied volumes/mounts and declare security contexts so they can land in PSS-restricted namespaces.
The design is solid: changes are purely additive, each new field is +optional with a pointer or slice, JSON tags are camelCase, and the embedded corev1.* types are the right shape for these semantics. Generated artifacts (zz_generated.deepcopy.go, CRDs) are included. Tests cover the meaningful paths (append, reserved-name rejection, duplicate rejection, FSGroup preservation/override, container SC replacement). Below are documentation clarifications and a few non-blocking suggestions.
Findings
Documentation accuracy
-
api/v1alpha1/task_types.go:103-108— ThePodSecurityContextdoc says "fields set here override Kelos defaults; fields left unset retain Kelos defaults". The implementation only carries forwardFSGroup(internal/controller/job_builder.go:621-630); every other field is taken verbatim from the user struct. This happens to be correct today only becauseFSGroupis the only field Kelos defaults (job_builder.go:365-367). If a future change adds another defaulted field on the pod SC, the doc will silently become wrong. Suggest restating the contract narrowly:PodSecurityContextreplaces the pod-level security context. As a carve-out, iffsGroupis unset here, Kelos's defaultfsGroupis preserved so the agent user retains read/write access to the workspace volume. All other fields are taken as-is from this struct. -
api/v1alpha1/task_types.go:96-101—VolumeMountssays names must reference a user-supplied volume "fromVolumesor a Kelos-managed volume (workspace,kelos-plugin)". That is documenting the implementation, butkelos-pluginis only present when plugin support is active for the Task; mounting it fromPodOverrides.VolumeMountswill succeed for Tasks that have plugins and fail (at pod admission) for those that don't. Worth either calling that out or recommending users only reference their ownvolumesentries.
Validation surface
-
internal/controller/job_builder.go:610-619— Reserved-name and duplicate-name validation runs atBuild()time, so users find out about the problem only when the Job materialises (status moves to a controller error, not an apply-time rejection). Since the Task spec is immutable post-create, this is a one-shot UX cost, but consider a CRD-level CEL rule, e.g.:// +kubebuilder:validation:XValidation:rule="!self.exists(v, v.name == 'workspace' || v.name == 'kelos-plugin')",message="podOverrides.volumes name must not be 'workspace' or 'kelos-plugin'" // +kubebuilder:validation:XValidation:rule="size(self.map(v, v.name)) == size(self)",message="podOverrides.volumes names must be unique"on the field. This gives operators feedback at
kubectl applyrather than at reconcile, and keeps the reserved-name list visible in the API surface (today it is hidden injob_builder.go:756-760). -
There is no controller-side check that each
VolumeMounts[].nameresolves to either a reserved volume or one ofPodOverrides.Volumes. The kubelet/apiserver will reject the pod, but a clearer pre-flight error would shorten the debugging loop. Optional.
List semantics (minor)
api/v1alpha1/task_types.go:94, 101—[]corev1.Volumeand[]corev1.VolumeMountare emitted as atomic lists in the CRD (internal/manifests/install-crd.yaml:1097, 3009), whereas upstreamPodSpec.Volumesis+listType=map +listMapKey=name. ForTaskthis is a no-op (spec is immutable), but forTaskSpawner.spec.taskTemplate.podOverridesit means any SSA touchingvolumesreplaces the whole list rather than merging by name. Adding the markers is the more conventional shape; matching the existingEnv []corev1.EnvVarstyle (also atomic) is acceptable too — flagging only for awareness, not as a blocker.
containerSecurityContext semantics
internal/controller/job_builder.go:632-634— When set, this completely replaces the agent container'sSecurityContext. The agent container has no Kelos-set defaults today, so "replace" matches "set". That is fine, but document it explicitly on the field so future contributors don't accidentally introduce a Kelos default and silently lose it (mirror the wording used onPodSecurityContext).
Field-by-field convention check
- All four fields are camelCase JSON, marked
+optional, use pointer/slice types, and have godoc — good. - No raw primitives that should be
resource.Quantity/metav1.Duration/metav1.Time— embeddedcorev1types handle those internally. - Prefix naming (
podSecurityContext,containerSecurityContext) is justified to disambiguate at the same struct level. - Backwards compatible: existing clients ignoring the four new fields see the same generated Job they would today.
Suggestions (optional)
- Consider
+kubebuilder:validation:MaxItemsonVolumesandVolumeMountsto bound API blast radius (small numbers — most realistic use cases need < 10). - The reserved-volume-name list (
workspace,kelos-plugin) is referenced in three places: the field godoc, the validator (job_builder.go:758-760), and tests. Consider exportingReservedVolumeNamesfrom the controller package (or moving it to theapipackage) so the list is single-sourced — handy if more managed volumes get added later. - The
internal/helmchart/render.gofix (replacingstrings.Split(..., "---\n")withyamlutil.NewYAMLReader) is the right call and not strictly part of the API change, but is well-motivated by the new CRD content. Nice catch.
/kelos needs-input
There was a problem hiding this comment.
🤖 Kelos Reviewer Agent @gjkim42
Review Summary
Verdict: APPROVE
Overall correctness: patch is correct
Scope: Adds volumes, volumeMounts, podSecurityContext, and containerSecurityContext to PodOverrides so spawned Task pods can attach extra volumes/mounts and declare security contexts; also fixes a latent helmchart-render YAML splitter that the new corev1 types tripped.
Findings Overview
| Priority | Count | File:Line | Summary |
|---|---|---|---|
| P0 | 0 | — | none |
| P1 | 0 | — | none |
| P2 | 1 | api/v1alpha1/task_types.go:110-115 | PSS-restricted goal is not achievable for tasks with a workspace or AgentConfig plugins/skills because init containers don't satisfy restricted rules |
| P3 | 2 | api/v1alpha1/task_types.go:103-108 | PodSecurityContext doc overstates the carve-out — only FSGroup is preserved |
| P3 | api/v1alpha1/task_types.go:96-101 | VolumeMounts doc lists kelos-plugin as a reserved mount, but it only exists for tasks with plugins/skills |
Findings
Documentation accuracy (Conventions)
- [P3]
api/v1alpha1/task_types.go:103-108—PodSecurityContext's godoc reads "fields set here override Kelos defaults; fields left unset retain Kelos defaults". The implementation ininternal/controller/job_builder.go:621-630only carriesFSGroupforward; every other field is taken verbatim from the user struct. This happens to be accurate today only becauseFSGroupis the sole field Kelos defaults (job_builder.go:365-367); if a future change defaults another pod-SC field, the doc silently becomes wrong. Restate the carve-out narrowly so the contract is explicit, e.g. "Replaces the pod-level security context. As a carve-out, whenfsGroupis unset here Kelos's defaultfsGroupis preserved so the agent user retains workspace access; all other fields are taken as-is." - [P3]
api/v1alpha1/task_types.go:96-101—VolumeMountsdoc says names must reference a user-supplied volume "fromVolumesor a Kelos-managed volume (workspace,kelos-plugin)".kelos-pluginonly exists when the Task'sAgentConfigdeclares plugins or skills (job_builder.go:508-519); referencing it from a Task without those will succeed at validation and fail at pod admission. Either dropkelos-pluginfrom the user-facing doc or add the conditional.
PSS-restricted scope (Correctness/Documentation)
- [P2]
api/v1alpha1/task_types.go:110-115andinternal/controller/job_builder.go:632-634—ContainerSecurityContextonly applies to the agent container, but PSS-restricted is enforced on every container in the pod, including init containers. The pre-existing init containersgit-clone(job_builder.go:388-397),plugin-setup(job_builder.go:527-535), andskills-install(job_builder.go:543-553) set at mostRunAsUserand do not setallowPrivilegeEscalation=falseorcapabilities.drop=[ALL]. The result is that for any Task with a workspace or with an AgentConfig that includes plugins/skills — i.e., the common case — the spawned pod will still be rejected by a PSS-restricted namespace despite the user settingcontainerSecurityContext. The headline rationale of the PR ("the pod can land in a PSS-restricted namespace") therefore only holds for Tasks with no workspace and no plugin/skill setup. Either apply restricted-compatible defaults to the init containers (allowPrivilegeEscalation=false,capabilities.drop=[ALL],readOnlyRootFilesystem=true) — they are all simple shell entrypoints that don't need privileged capabilities — or document this limitation in the new field's godoc and indocs/reference.mdso operators understand the boundary.
Suggestions (optional)
- [P3] Reserved-name validation runs in
JobBuilder.Build(), so a bad volume name surfaces as a controller reconcile error rather than akubectl applyrejection. A CRD-level CEL rule onpodOverrides.volumes(name not in ['workspace','kelos-plugin']and uniqueness byname) would shorten the feedback loop and put the reserved-name list in the API surface; the validator can stay as a defense-in-depth layer. - [P3] Consider exporting the reserved-volume-name set or moving it to the
apipackage so the godoc, the validator (job_builder.go:756-760), and the tests all reference one source — handy if more managed volumes are added. - [P3] The
splitYAMLDocsswitch fromstrings.Split(content, "---\n")toyamlutil.NewYAMLReaderis the correct fix and is well-motivated by the newPodSecurityContext.fsGroupdescription, which contains the literalrw-rw----\n(4 dashes + newline) that the naive split treated as a document boundary. Nice catch.
Key takeaways
- Implementation, deepcopy, CRDs, and tests are coherent and the YAML-split fix is correct.
- The PSS-restricted promise in the PR description is partially broken for Tasks that have a workspace or a plugin/skill-bearing AgentConfig; tightening the godoc or extending the security context to init containers would close the gap.
What type of PR is this?
/kind feature
What this PR does / why we need it:
Adds four optional fields to the shared
PodOverridesstruct so that spawned Task pods can attach extra volumes, mount them into the agent container, and declare pod- and container-level security contexts:volumes— appended to the Job'sPodSpec.Volumes. User-supplied names must not collide with Kelos-managed names (workspace,kelos-plugin); duplicates are rejected.volumeMounts— appended to the agent container's mounts. A flat list rather than a map keyed by container name; only the agent container is exposed, init containers stay private.podSecurityContext— replaces the pod-level security context set by the controller. As a carve-out,fsGroupis preserved from Kelos's default when the user does not set it explicitly, so the agent user retains read/write access to the workspace volume.containerSecurityContext— replaces the agent container's security context, so callers can declareallowPrivilegeEscalation: false,capabilities.drop: [ALL],readOnlyRootFilesystem: true, etc., and the pod can land in a PSS-restricted namespace.Without these fields, namespaces enforcing PSS
restrictedreject Kelos-spawned pods because the spawned containers don't declare the required security primitives — so operators are forced to relax the agent namespace tobaseline. With these fields exposed, callers can opt back intorestrictedand can also mount things like writable profile dirs, custom CA bundles, or per-task config files.This PR also fixes a latent bug in
internal/helmchart/render.go: rendered templates were being split into YAML documents withstrings.Split(content, "---\n"), which corrupts documents whose description text happens to contain"---\n". The newcorev1types pulled into the CRD surface this —PodSecurityContext.fsGroup's description contains the literal examplerw-rw----\n, which the naive split treats as a document boundary. Replaced withk8s.io/apimachinery/pkg/util/yaml's reader, which only splits on lines that exactly match---. A couple of render tests that relied on a substring search for":latest"were also tightened to match real image references rather than CRD description text mentioning:latestin prose.Which issue(s) this PR is related to:
N/A
Special notes for your reviewer:
ServiceAccountNamewas already exposed onPodOverrides, so this PR only adds the four new fields above.VolumeMountsandContainerSecurityContextare intentionally not keyed by container name — the agent container is the only meaningful target, and exposing init containers would let users break the workspace setup. The flat shape mirrors howResourcesis exposed today.PodSecurityContextmerge semantics: user values win field-by-field on the user-supplied struct (Kelos copies it viaDeepCopyand applies it directly), with one exception — if the user did not setFSGroup, the controller preserves the Kelos-defaultFSGroup, since dropping it silently breaks workspace volume access.Does this PR introduce a user-facing change?