From db889a1a803e4ab09f495d9965fd820545563c4e Mon Sep 17 00:00:00 2001 From: Omer Aplatony Date: Sun, 29 Dec 2024 12:10:30 +0200 Subject: [PATCH 1/7] Document VPA behavior for sidecar containers and how to change it Signed-off-by: Omer Aplatony --- .../docs/sidecar-containers.md | 173 ++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 vertical-pod-autoscaler/docs/sidecar-containers.md diff --git a/vertical-pod-autoscaler/docs/sidecar-containers.md b/vertical-pod-autoscaler/docs/sidecar-containers.md new file mode 100644 index 000000000000..4f3ca013506f --- /dev/null +++ b/vertical-pod-autoscaler/docs/sidecar-containers.md @@ -0,0 +1,173 @@ +# VPA Sidecar Container Management + +The Vertical Pod Autoscaler (VPA) has specific behavior when dealing with sidecar containers that are injected into pods via admission webhooks. This document explains the default behavior and how to customize it for your needs. + +## Understanding VPA and Container Policies + +### Default Container Policies + +To understand why sidecar handling is important, let's first look at how VPA manages containers by default. This default behavior is at the root of why special consideration is needed for sidecars. + +When you create a VPA resource for a pod, it automatically attempts to manage ALL containers in that pod - not just the ones you explicitly configure. This happens because: + +1. VPA applies a default `containerPolicy` with `mode: Auto` to any container not explicitly configured +2. This automatic inclusion means VPA will try to manage resources for every container it sees +3. Even sidecars injected into the pod will fall under VPA's management unless special steps are taken + +This default "manage everything" approach can cause problems with sidecars because: +- Sidecar containers often have their own resource requirements set by their injection webhooks +- VPA's automatic management may conflict with these requirements +- Without proper handling, this can lead to resource conflicts and pod instability + +Example of a VPA resource that explicitly configures one container but affects all: + +```yaml +apiVersion: autoscaling.k8s.io/v1 +kind: VerticalPodAutoscaler +metadata: + name: my-app-vpa +spec: + targetRef: + apiVersion: "apps/v1" + kind: Deployment + name: my-app + updatePolicy: + updateMode: "Auto" + resourcePolicy: + containerPolicies: + - containerName: main-container + minAllowed: + cpu: 100m + memory: 50Mi +# Note: Other containers will still get the default Auto mode +``` + +## Default Behavior: Ignoring Sidecar Containers + +### The vpaObservedContainers Annotation + +VPA uses a special annotation to track which containers were present in the pod before any webhook injections: + +```yaml +metadata: + annotations: + vpaObservedContainers: "main-container,logging-sidecar" +``` + +This annotation is crucial because: +1. It's added by the VPA admission controller webhook +2. Only containers listed in this annotation will be managed by VPA +3. The annotation must be added before sidecar injection for the default behavior to work + +### Webhook Ordering Importance + +The order of webhook execution is determined alphabetically by webhook names. For example: + +```yaml +webhooks: +- name: a.sidecar-injector.kubernetes.io # Executes first +- name: b.vpa.kubernetes.io # Executes second +- name: c.another-sidecar.kubernetes.io # Executes third +``` + +### The Eviction Loop Problem + +Without proper handling of sidecar containers, the following problematic sequence could occur: + +1. VPA admission controller sets resources for all containers +2. Sidecar webhook injects a new container with its own resource requirements +3. The pod starts with mismatched resources +4. VPA detects the mismatch and evicts the pod +5. The process repeats, creating an endless loop + +## Customizing VPA Behavior for Sidecar Containers + +### Option 1: Webhook Ordering + +To have VPA manage sidecar resources, ensure your webhook names follow this pattern: + +```yaml +webhooks: +- name: sidecar-injector.kubernetes.io # Executes first +- name: zz.vpa.kubernetes.io # Executes last +``` + +This ensures: +1. Sidecars are injected first +2. VPA sees the complete pod with all sidecars +3. The `vpaObservedContainers` annotation includes all containers + +### Option 2: Webhook Reinvocation + +Configure the VPA webhook to reinvoke after sidecar injection: + +```yaml +apiVersion: admissionregistration.k8s.io/v1 +kind: MutatingWebhookConfiguration +metadata: + name: vpa-webhook-config +webhooks: +- name: vpa.kubernetes.io + reinvocationPolicy: IfNeeded + rules: + - apiGroups: [""] + apiVersions: ["v1"] + operations: ["CREATE", "UPDATE"] + resources: ["pods"] +``` + +This configuration: +1. Allows the VPA webhook to be called multiple times +2. Ensures resource recommendations are applied after all sidecars are injected +3. Prevents the eviction loop problem + +## Implementation Examples + +### Example 1: Istio Sidecar Integration + +When using Istio, which injects proxy sidecars: + +```yaml +# Istio sidecar injector webhook +apiVersion: admissionregistration.k8s.io/v1 +kind: MutatingWebhookConfiguration +metadata: + name: istio-sidecar-injector +webhooks: +- name: a.sidecar-injector.istio.io # Alphabetically first + +# VPA webhook configuration +apiVersion: admissionregistration.k8s.io/v1 +kind: MutatingWebhookConfiguration +metadata: + name: vpa-webhook-config +webhooks: +- name: z.vpa.kubernetes.io # Alphabetically last + reinvocationPolicy: IfNeeded +``` + +### Example 2: Custom VPA Configuration with Sidecars + +```yaml +apiVersion: autoscaling.k8s.io/v1 +kind: VerticalPodAutoscaler +metadata: + name: web-app-vpa +spec: + targetRef: + apiVersion: "apps/v1" + kind: Deployment + name: web-app + updatePolicy: + updateMode: "Auto" + resourcePolicy: + containerPolicies: + - containerName: main-container + minAllowed: + cpu: 100m + memory: 128Mi + - containerName: logging-sidecar # Explicitly configure sidecar + minAllowed: + cpu: 50m + memory: 64Mi +``` From 989067050a20cc3bac88b47aed792a990ee25978 Mon Sep 17 00:00:00 2001 From: Omer Aplatony Date: Mon, 30 Dec 2024 16:16:59 +0200 Subject: [PATCH 2/7] specify pod kind Co-authored-by: Adrian Moisey --- vertical-pod-autoscaler/docs/sidecar-containers.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/vertical-pod-autoscaler/docs/sidecar-containers.md b/vertical-pod-autoscaler/docs/sidecar-containers.md index 4f3ca013506f..9e74915c7103 100644 --- a/vertical-pod-autoscaler/docs/sidecar-containers.md +++ b/vertical-pod-autoscaler/docs/sidecar-containers.md @@ -49,6 +49,8 @@ spec: VPA uses a special annotation to track which containers were present in the pod before any webhook injections: ```yaml +apiVersion: v1 +kind: Pod metadata: annotations: vpaObservedContainers: "main-container,logging-sidecar" From 3e4adddc3a8060f0821bf3825990d04fbc86c0b3 Mon Sep 17 00:00:00 2001 From: Omer Aplatony Date: Mon, 30 Dec 2024 16:26:15 +0200 Subject: [PATCH 3/7] docs: clarify sidecar container definition and link to k8s pattern Signed-off-by: Omer Aplatony --- vertical-pod-autoscaler/docs/sidecar-containers.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/vertical-pod-autoscaler/docs/sidecar-containers.md b/vertical-pod-autoscaler/docs/sidecar-containers.md index 9e74915c7103..634a21bbebf9 100644 --- a/vertical-pod-autoscaler/docs/sidecar-containers.md +++ b/vertical-pod-autoscaler/docs/sidecar-containers.md @@ -1,6 +1,8 @@ # VPA Sidecar Container Management -The Vertical Pod Autoscaler (VPA) has specific behavior when dealing with sidecar containers that are injected into pods via admission webhooks. This document explains the default behavior and how to customize it for your needs. +In this document, "sidecar container" refers to any additional container that isn't the main application container in a pod. This is distinct from the formal [Kubernetes sidecar pattern](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/), which describes a specific design pattern where a container enhances or extends the main container's functionality. Our usage here applies to all additional containers, regardless of their architectural pattern or purpose. + +The Vertical Pod Autoscaler (VPA) has specific behavior when dealing with these additional containers that are injected into pods via admission webhooks. ## Understanding VPA and Container Policies From a6437956ce16e6f2dcd232d0529eba149ed4483f Mon Sep 17 00:00:00 2001 From: Omer Aplatony Date: Fri, 10 Jan 2025 13:45:56 +0200 Subject: [PATCH 4/7] Update vertical-pod-autoscaler/docs/sidecar-containers.md Co-authored-by: Marco Voelz --- vertical-pod-autoscaler/docs/sidecar-containers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vertical-pod-autoscaler/docs/sidecar-containers.md b/vertical-pod-autoscaler/docs/sidecar-containers.md index 634a21bbebf9..2fcd5226064b 100644 --- a/vertical-pod-autoscaler/docs/sidecar-containers.md +++ b/vertical-pod-autoscaler/docs/sidecar-containers.md @@ -19,7 +19,7 @@ When you create a VPA resource for a pod, it automatically attempts to manage AL This default "manage everything" approach can cause problems with sidecars because: - Sidecar containers often have their own resource requirements set by their injection webhooks - VPA's automatic management may conflict with these requirements -- Without proper handling, this can lead to resource conflicts and pod instability +- Without proper handling, this can lead to problems like endless eviction loops Example of a VPA resource that explicitly configures one container but affects all: From 5504fc5e1d5feec14d4a46ef6927c68d7eb3e2e9 Mon Sep 17 00:00:00 2001 From: Omer Aplatony Date: Fri, 10 Jan 2025 13:46:12 +0200 Subject: [PATCH 5/7] Update vertical-pod-autoscaler/docs/sidecar-containers.md Co-authored-by: Marco Voelz --- vertical-pod-autoscaler/docs/sidecar-containers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vertical-pod-autoscaler/docs/sidecar-containers.md b/vertical-pod-autoscaler/docs/sidecar-containers.md index 2fcd5226064b..03e2b960ca78 100644 --- a/vertical-pod-autoscaler/docs/sidecar-containers.md +++ b/vertical-pod-autoscaler/docs/sidecar-containers.md @@ -79,7 +79,7 @@ webhooks: Without proper handling of sidecar containers, the following problematic sequence could occur: 1. VPA admission controller sets resources for all containers -2. Sidecar webhook injects a new container with its own resource requirements +2. Sidecar webhook reconciles the injected sidecar container to the original resource requirements 3. The pod starts with mismatched resources 4. VPA detects the mismatch and evicts the pod 5. The process repeats, creating an endless loop From e04536d580b6b04e706b876a3ae9e318e1371026 Mon Sep 17 00:00:00 2001 From: Omer Aplatony Date: Fri, 10 Jan 2025 13:46:37 +0200 Subject: [PATCH 6/7] Update vertical-pod-autoscaler/docs/sidecar-containers.md Co-authored-by: Marco Voelz --- vertical-pod-autoscaler/docs/sidecar-containers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vertical-pod-autoscaler/docs/sidecar-containers.md b/vertical-pod-autoscaler/docs/sidecar-containers.md index 03e2b960ca78..0343179292ce 100644 --- a/vertical-pod-autoscaler/docs/sidecar-containers.md +++ b/vertical-pod-autoscaler/docs/sidecar-containers.md @@ -88,7 +88,7 @@ Without proper handling of sidecar containers, the following problematic sequenc ### Option 1: Webhook Ordering -To have VPA manage sidecar resources, ensure your webhook names follow this pattern: +If you know that you only use sidecar injecting webhooks which _don't_ reconcile Container resources, you can choose to have VPA manage sidecar resources. Ensure your webhook names follow this pattern, resulting in the VPA admission-controller webhook to be executed last: ```yaml webhooks: From 3156829a20d6322dc7513b635f1b3f65753a93bb Mon Sep 17 00:00:00 2001 From: Omer Aplatony Date: Fri, 10 Jan 2025 13:47:00 +0200 Subject: [PATCH 7/7] Update vertical-pod-autoscaler/docs/sidecar-containers.md Co-authored-by: Marco Voelz --- vertical-pod-autoscaler/docs/sidecar-containers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vertical-pod-autoscaler/docs/sidecar-containers.md b/vertical-pod-autoscaler/docs/sidecar-containers.md index 0343179292ce..281b84f2ec16 100644 --- a/vertical-pod-autoscaler/docs/sidecar-containers.md +++ b/vertical-pod-autoscaler/docs/sidecar-containers.md @@ -1,6 +1,6 @@ # VPA Sidecar Container Management -In this document, "sidecar container" refers to any additional container that isn't the main application container in a pod. This is distinct from the formal [Kubernetes sidecar pattern](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/), which describes a specific design pattern where a container enhances or extends the main container's functionality. Our usage here applies to all additional containers, regardless of their architectural pattern or purpose. +In this document, "sidecar container" refers to any additional Container that isn't the main application Container in a Pod. This is distinct from the [native Kubernetes sidecar pattern](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/), which makes use of `initContainers`. Our usage here applies to all additional regular `containers` only, as VPA does not support `initContainers` yet. The Vertical Pod Autoscaler (VPA) has specific behavior when dealing with these additional containers that are injected into pods via admission webhooks.