[agent-local]: add ephemeral volume and StatefulSet support for cache/data directories

[Boniker]

Issue

The chart supports two storage modes for persistence.cache and persistence.data:

• emptyDir - cache lost on every pod restart
• PersistentVolumeClaim - single PVC shared across all pods, it's incompatible with replicaCount > 1 and ReadWriteOnce since only one node can mount at a time
• ReadWriteMany (like efs) - works with multiple replicas but adds significant cost due to provisioned throughput requirements

Neither option works correctly for multi-replica deployments. Without native support we patch the chart's rendered cache-dir volume via a kustomize JSON patch at a hardcoded volume index - fragile and breaks native Helm workflows.

Opportunities

Generic ephemeral volumes

Generic ephemeral volumes provision a dedicated PVC per pod automatically, tied to the pod lifecycle. Each agent gets its own cache that persists across container restarts but is cleaned up on pod termination.

StatefulSet support

A StatefulSet with volumeClaimTemplates is the canonical Kubernetes pattern for stateful workloads needing per-pod persistent storage.

[mermoldy]

Hi @Boniker, thanks for the detailed write-up, having to patch the rendered volume at a hardcoded index isn't something you should have to maintain.

We'll evaluate the two approaches together with the migration path for existing installations - the main open question on our side is how to introduce the change without disrupting current disk configurations. We'll prioritize this internally and follow up here once we've settled on a direction.