Generalized OpenShift Sandboxed Containers Webhook Proposal

[c3d]

Note: This builds on top of PR #1058.

This proposal was written with the help of claude-4.5-opus-high, and despite some quick review, may contain AI-typical flaws 😄

Executive Summary

This proposal outlines the design for a generalized mutating admission webhook for the OpenShift Sandboxed Containers (OSC) Operator. Building on the existing memory overhead annotation work, this webhook will provide a unified, configurable solution for:

1. Injecting Kata-specific pod annotations (memory, CPU, initdata, etc.)
2. Automatically adding runtimeClass to pods in designated namespaces
3. Supporting Confidential Containers (CoCo) baremetal requirements
4. Providing an opt-out mechanism for customers using alternative solutions (OPA, Kyverno)

Background

Previous Discussions

The need for an OSC mutating webhook has come up several times in earlier discussions. The primary driver was to automatically add the runtime class to pods based on some criteria. Until now, users were redirected to use their own mutating webhooks (OPA policy, Kyverno policy, etc.).

Current Drivers

With recent Confidential Containers (CoCo) work, the need for an OSC-specific webhook has become more pressing:

• Initdata injection: CoCo baremetal pods require initdata annotations
• Memory overhead: Prevent host cgroup OOM by injecting memory overhead annotations
• CPU configuration: Pass CPU-related configurations to the Kata runtime
• RuntimeClass injection: Automatically set runtimeClassName for pods in specific namespaces

Existing Work

1. Memory Overhead Annotation (this operator): Injects io.katacontainers.config.hypervisor.memory_overhead for Kata pods. See MEMORY_OVERHEAD_ANNOTATION_PROPOSAL.md (in controller: Add overhead annotation using pod mutating webhook #1058).

2. Peer Pods Webhook (cloud-api-adaptor): Handles extended resources for peer pods. This implementation can serve as a reference for resource-based mutations.

Requirements

Functional Requirements

ID	Requirement	Priority
FR1	Inject Kata pod annotations (memory_overhead, default_vcpus, etc.)	High
FR2	Inject initdata annotation for CoCo baremetal pods	High
FR3	Automatically add runtimeClassName to pods in specific namespaces	Medium
FR4	Support multiple runtime classes (kata, kata-remote, kata-cc, etc.)	High
FR5	Ability to disable webhook via KataConfig	High
FR6	Configurable annotation rules per namespace or label selector	Medium
FR7	Preserve existing pod annotations (no overwrite unless configured)	High

Non-Functional Requirements

ID	Requirement	Priority
NFR1	Webhook latency < 50ms per request	High
NFR2	Graceful degradation if webhook is unavailable	High
NFR3	Comprehensive audit logging	Medium
NFR4	Compatible with OPA/Kyverno (can coexist or be disabled)	High

Proposed Architecture

High-Level Design

┌─────────────────────────────────────────────────────────────────┐
│                     Kubernetes API Server                       │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼ Pod Create/Update
┌─────────────────────────────────────────────────────────────────┐
│                   OSC Mutating Webhook                          │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │                  Mutation Pipeline                         │ │
│  │  ┌──────────────┐ ┌──────────────┐ ┌─────────────────────┐ │ │
│  │  │ RuntimeClass │ │  Annotation  │ │     Resource        │ │ │
│  │  │  Injector    │→│  Injector    │→│     Mutator         │ │ │
│  │  └──────────────┘ └──────────────┘ └─────────────────────┘ │ │
│  └────────────────────────────────────────────────────────────┘ │
│                              │                                  │
│                              ▼                                  │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │                  Configuration Sources                     │ │
│  │  ┌──────────────┐ ┌──────────────┐ ┌─────────────────────┐ │ │
│  │  │  KataConfig  │ │  Namespace   │ │    WebhookConfig    │ │ │
│  │  │    CRD       │ │  Labels      │ │       CRD           │ │ │
│  │  └──────────────┘ └──────────────┘ └─────────────────────┘ │ │
│  └────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘

Mutation Pipeline

The webhook processes pods through a pipeline of mutators:

1. RuntimeClass Injector: Adds runtimeClassName based on namespace/label rules
2. Annotation Injector: Adds Kata-specific annotations based on KataConfig
3. Resource Mutator: Adjusts resource requests/limits if needed (like peer pods webhook)

Configuration Model

Option 1: Extend KataConfig (Recommended)

type KataConfigSpec struct {
    // ... existing fields ...

    // Webhook configures the OSC mutating webhook behavior
    // +optional
    Webhook *WebhookConfig `json:"webhook,omitempty"`
}

type WebhookConfig struct {
    // Enabled controls whether the webhook is active
    // +optional
    // +kubebuilder:default:=true
    Enabled *bool `json:"enabled,omitempty"`

    // FailurePolicy determines behavior when webhook fails
    // +optional
    // +kubebuilder:default:="Fail"
    // +kubebuilder:validation:Enum=Fail;Ignore
    FailurePolicy string `json:"failurePolicy,omitempty"`

    // Annotations defines annotation injection rules
    // +optional
    Annotations *AnnotationConfig `json:"annotations,omitempty"`

    // RuntimeClassInjection defines rules for automatic runtimeClass injection
    // +optional
    RuntimeClassInjection *RuntimeClassInjectionConfig `json:"runtimeClassInjection,omitempty"`
}

type AnnotationConfig struct {
    // MemoryOverheadMB specifies memory overhead for Kata pods
    // +optional
    // +kubebuilder:default:=350
    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:Maximum=2048
    MemoryOverheadMB *int32 `json:"memoryOverheadMB,omitempty"`

    // DefaultVCPUs specifies default vCPU count
    // +optional
    DefaultVCPUs *int32 `json:"defaultVCPUs,omitempty"`

    // Initdata specifies initdata for CoCo pods
    // +optional
    Initdata *string `json:"initdata,omitempty"`

    // Custom allows arbitrary annotation injection
    // +optional
    Custom map[string]string `json:"custom,omitempty"`
}

type RuntimeClassInjectionConfig struct {
    // Enabled controls whether runtimeClass injection is active
    // +optional
    // +kubebuilder:default:=false
    Enabled *bool `json:"enabled,omitempty"`

    // DefaultRuntimeClass is the runtimeClass to inject
    // +optional
    // +kubebuilder:default:="kata"
    DefaultRuntimeClass string `json:"defaultRuntimeClass,omitempty"`

    // NamespaceSelector selects namespaces for injection
    // +optional
    NamespaceSelector *metav1.LabelSelector `json:"namespaceSelector,omitempty"`

    // PodSelector selects pods for injection
    // +optional
    PodSelector *metav1.LabelSelector `json:"podSelector,omitempty"`

    // ExcludeNamespaces lists namespaces to exclude
    // +optional
    ExcludeNamespaces []string `json:"excludeNamespaces,omitempty"`
}

Option 2: Separate WebhookConfig CRD

Create a dedicated OSCWebhookConfig CRD for more complex scenarios:

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
type OSCWebhookConfig struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   OSCWebhookConfigSpec   `json:"spec,omitempty"`
    Status OSCWebhookConfigStatus `json:"status,omitempty"`
}

type OSCWebhookConfigSpec struct {
    // Rules defines mutation rules
    Rules []MutationRule `json:"rules,omitempty"`
}

type MutationRule struct {
    // Name is a unique identifier for this rule
    Name string `json:"name"`

    // Match defines when this rule applies
    Match RuleMatch `json:"match"`

    // Mutate defines what mutations to apply
    Mutate RuleMutate `json:"mutate"`
}

type RuleMatch struct {
    // Namespaces to match (empty = all)
    Namespaces []string `json:"namespaces,omitempty"`

    // NamespaceSelector for label-based matching
    NamespaceSelector *metav1.LabelSelector `json:"namespaceSelector,omitempty"`

    // PodSelector for pod label matching
    PodSelector *metav1.LabelSelector `json:"podSelector,omitempty"`

    // RuntimeClasses to match (empty = all Kata classes)
    RuntimeClasses []string `json:"runtimeClasses,omitempty"`
}

type RuleMutate struct {
    // RuntimeClassName to set (if empty, don't set)
    RuntimeClassName string `json:"runtimeClassName,omitempty"`

    // Annotations to add
    Annotations map[string]string `json:"annotations,omitempty"`

    // Labels to add
    Labels map[string]string `json:"labels,omitempty"`
}

Implementation Details

Generalized PodMutator

type PodMutator struct {
    Client          client.Client
    Log             logr.Logger
    webhookEnabled  bool
    mutators        []PodMutatorFunc
}

type PodMutatorFunc func(ctx context.Context, pod *corev1.Pod, config *WebhookConfig) error

func NewPodMutator(mgr ctrl.Manager) *PodMutator {
    m := &PodMutator{
        Client: mgr.GetClient(),
        Log:    log.Log.WithName("pod-mutator"),
    }

    // Register mutators in order
    m.mutators = []PodMutatorFunc{
        m.injectRuntimeClass,
        m.injectMemoryOverhead,
        m.injectCPUConfig,
        m.injectInitdata,
        m.injectCustomAnnotations,
    }

    return m
}

func (m *PodMutator) Default(ctx context.Context, obj runtime.Object) error {
    pod, ok := obj.(*corev1.Pod)
    if !ok {
        return fmt.Errorf("expected a Pod but got a %T", obj)
    }

    // Get webhook configuration
    config, err := m.getWebhookConfig(ctx)
    if err != nil {
        return err
    }

    // Check if webhook is enabled
    if config.Webhook != nil && config.Webhook.Enabled != nil && !*config.Webhook.Enabled {
        m.Log.V(1).Info("webhook disabled, skipping mutations")
        return nil
    }

    // Run mutation pipeline
    for _, mutator := range m.mutators {
        if err := mutator(ctx, pod, config.Webhook); err != nil {
            m.Log.Error(err, "mutation failed")
            return err
        }
    }

    return nil
}

RuntimeClass Injection

func (m *PodMutator) injectRuntimeClass(ctx context.Context, pod *corev1.Pod, config *WebhookConfig) error {
    if config == nil || config.RuntimeClassInjection == nil {
        return nil
    }

    injection := config.RuntimeClassInjection
    if injection.Enabled == nil || !*injection.Enabled {
        return nil
    }

    // Skip if pod already has a runtimeClass
    if pod.Spec.RuntimeClassName != nil {
        return nil
    }

    // Check namespace selector
    if !m.matchesNamespaceSelector(ctx, pod.Namespace, injection.NamespaceSelector) {
        return nil
    }

    // Check pod selector
    if !m.matchesPodSelector(pod, injection.PodSelector) {
        return nil
    }

    // Check exclusions
    for _, ns := range injection.ExcludeNamespaces {
        if pod.Namespace == ns {
            return nil
        }
    }

    // Inject runtimeClass
    runtimeClass := injection.DefaultRuntimeClass
    if runtimeClass == "" {
        runtimeClass = "kata"
    }
    pod.Spec.RuntimeClassName = &runtimeClass

    m.Log.Info("injected runtimeClass",
        "pod", pod.Name,
        "namespace", pod.Namespace,
        "runtimeClass", runtimeClass)

    return nil
}

CoCo Initdata Injection

const (
    KataInitdataAnnotation = "io.katacontainers.config.runtime.cc_init_data"
)

func (m *PodMutator) injectInitdata(ctx context.Context, pod *corev1.Pod, config *WebhookConfig) error {
    if config == nil || config.Annotations == nil {
        return nil
    }

    if config.Annotations.Initdata == nil || *config.Annotations.Initdata == "" {
        return nil
    }

    // Only inject for CoCo runtime classes
    if !m.isCoCoRuntimeClass(pod) {
        return nil
    }

    if pod.Annotations == nil {
        pod.Annotations = make(map[string]string)
    }

    pod.Annotations[KataInitdataAnnotation] = *config.Annotations.Initdata

    m.Log.Info("injected initdata annotation",
        "pod", pod.Name,
        "namespace", pod.Namespace)

    return nil
}

func (m *PodMutator) isCoCoRuntimeClass(pod *corev1.Pod) bool {
    if pod.Spec.RuntimeClassName == nil {
        return false
    }

    cocoClasses := []string{"kata-cc", "kata-cc-sim", "kata-remote"}
    for _, cc := range cocoClasses {
        if *pod.Spec.RuntimeClassName == cc {
            return true
        }
    }
    return false
}

Webhook Disable Feature

func (m *PodMutator) getWebhookConfig(ctx context.Context) (*kataconfigurationv1.KataConfig, error) {
    kataConfigs := &kataconfigurationv1.KataConfigList{}
    if err := m.Client.List(ctx, kataConfigs); err != nil {
        return nil, fmt.Errorf("failed to list KataConfigs: %w", err)
    }

    if len(kataConfigs.Items) == 0 {
        return nil, nil
    }

    // Return first KataConfig (typically only one exists)
    return &kataConfigs.Items[0], nil
}

Sample Configurations

Basic Configuration (Memory Overhead Only)

apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: example-kataconfig
spec:
  webhook:
    enabled: true
    annotations:
      memoryOverheadMB: 512

CoCo Baremetal Configuration

apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: coco-kataconfig
spec:
  webhook:
    enabled: true
    annotations:
      memoryOverheadMB: 512
      defaultVCPUs: 2
      initdata: "YWxnb3JpdGhtPXNuZXZlci..." # base64 encoded

RuntimeClass Injection for Namespace

apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: auto-kata-namespace
spec:
  webhook:
    enabled: true
    annotations:
      memoryOverheadMB: 350
    runtimeClassInjection:
      enabled: true
      defaultRuntimeClass: kata
      namespaceSelector:
        matchLabels:
          sandboxed-containers: "true"
      excludeNamespaces:
        - kube-system
        - openshift-*

Disable Webhook (Use OPA/Kyverno)

apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: external-policy
spec:
  webhook:
    enabled: false  # Customer uses OPA/Kyverno

Supported Annotations

Annotation	Description	Corresponding Config
io.katacontainers.config.hypervisor.memory_overhead	Memory overhead in MB	annotations.memoryOverheadMB
io.katacontainers.config.hypervisor.default_vcpus	Default vCPU count	annotations.defaultVCPUs
io.katacontainers.config.runtime.cc_init_data	CoCo initdata	annotations.initdata
io.katacontainers.config.hypervisor.default_memory	Default memory	annotations.custom
io.katacontainers.config.hypervisor.virtio_fs_cache	VirtioFS cache mode	annotations.custom
Custom	Any annotation	annotations.custom

Migration Path

Phase 1: Generalize Current Implementation

1. Refactor PodMutator to use mutation pipeline
2. Add WebhookConfig to KataConfigSpec
3. Add enabled flag (default: true for backward compatibility)
4. Maintain existing memory overhead behavior

Phase 2: Add New Mutations

1. Implement CPU annotation injection
2. Implement initdata injection for CoCo
3. Add custom annotation support

Phase 3: RuntimeClass Injection

1. Implement namespace/label-based runtimeClass injection
2. Add comprehensive testing
3. Documentation and examples

Phase 4: Advanced Features

1. Consider separate OSCWebhookConfig CRD for complex scenarios
2. Add metrics and observability
3. Performance optimization

Comparison with Alternatives

Feature	OSC Webhook	OPA/Gatekeeper	Kyverno
OSC-specific logic	✅ Native	❌ Custom policy	❌ Custom policy
KataConfig integration	✅ Direct	❌ External	❌ External
Learning curve	Low	High	Medium
Maintenance	Operator team	User	User
Flexibility	Medium	High	High
Can coexist	✅	✅	✅

Recommendation: Provide the OSC webhook with a disable option. Users with complex requirements or existing policy infrastructure can disable it and use OPA/Kyverno.

Security Considerations

1. RBAC: Minimal permissions (list KataConfigs, mutate pods)
2. Namespace exclusions: Always exclude system namespaces by default
3. Audit logging: Log all mutations for compliance
4. TLS: Webhook communication over TLS

Testing Strategy

1. Unit Tests: Each mutator function
2. Integration Tests: Full webhook pipeline
3. E2E Tests: Real cluster with all mutation scenarios
4. Compatibility Tests: Ensure coexistence with OPA/Kyverno

References

• Memory Overhead Annotation Proposal
• Peer Pods Webhook (cloud-api-adaptor)
• Kubernetes Admission Webhooks Best Practices
• Kyverno Policies
• OPA Gatekeeper

Conclusion

This proposal provides a path to evolve the current memory overhead webhook into a generalized OSC webhook that addresses multiple use cases while maintaining simplicity and providing an opt-out mechanism. The phased approach allows incremental delivery while building on proven existing work.

The key design decisions are:

1. Pipeline architecture: Composable, testable mutations
2. KataConfig integration: Single source of truth for configuration
3. Opt-out capability: Respects users with existing policy infrastructure
4. Backward compatibility: Existing deployments continue to work

[bpradipt]

How about webhook as a feature config which will provide us some flexibility with iteration instead of updating KataConfig?

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale