From a372a8fb73f52794ba9d13945e3433c2495b5a26 Mon Sep 17 00:00:00 2001 From: Shreeharsh Shinde Date: Sat, 31 Jan 2026 15:23:35 +0530 Subject: [PATCH 1/4] docs: add architecture.md Signed-off-by: Shreeharsh Shinde --- README.md | 1 + docs/architecture.md | 128 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 129 insertions(+) create mode 100644 docs/architecture.md diff --git a/README.md b/README.md index cf110a0c..7574b5c6 100644 --- a/README.md +++ b/README.md @@ -94,6 +94,7 @@ and how they can configure Tortoise so that they can let tortoises autoscale the - [Horizontal scaling](./docs/horizontal.md): describes how the Tortoise does the horizontal autoscaling internally. - [Vertical scaling](./docs/vertical.md): describes how the Tortoise does the vertical autoscaling internally. - [Technically details](./docs/internal.md): describes the technically details of Tortoise. (mostly for the contributors) +- [Architecture](./docs/architecture.md): describes the high-level architecture of Tortoise. - [Contributor guide](./docs/contributor-guide.md): describes other stuff for the contributor. (testing etc) ## API definition diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 00000000..0cee2b80 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,128 @@ +# Architecture + +This document provides a deep dive into the internal architecture of Tortoise. It is intended for contributors who want to understand how the controller works under the hood. + +## System Overview + +Tortoise creates a feedback loop between the user's workload (Deployment), the autoscalers (HPA, VPA), and the Tortoise optimization logic. + +```mermaid +graph TD + User[User] -->|Apply| Tortoise[Tortoise CR] + Tortoise -->|Reconcile| Controller[Tortoise Controller] + + subgraph "Managed Resources" + HPA[Horizontal Pod Autoscaler] + VPA[Vertical Pod Autoscaler] + end + + subgraph "Target Workload" + Deployment[Deployment] + ReplicaSet[ReplicaSet] + Pods[Pods] + end + + Controller -- 1. Creates/Updates --> HPA + Controller -- 2. Creates/Updates (Monitor Mode) --> VPA + + VPA -- 3. Analyzes Usage --> Pods + VPA -.->|Recommendation| Controller + + Controller -- 4. Calculates Optimization --> Recommendations[Internal Recommendations] + + Recommendations -- 5a. Updates Min/Max/Targets --> HPA + Recommendations -- 5b. Vertical Scaling (Rolling Update) --> Deployment + HPA -- 5c. Scales Replicas --> Deployment +``` + +## Core Components + +### 1. Tortoise Controller +The heart of the system. It runs a reconciliation loop that ensures the actual state matches the desired state defined in the `Tortoise` Custom Resource (CR). + +**Key Responsibilities:** +- **Orchestration**: Manages the lifecycle of associated HPA and VPA resources. +- **Optimization**: Calculates ideal resource requests and auto-scaling parameters based on historical data (from VPA) and current usage. +- **Actuation**: Applies changes to HPA (Horizontal scaling) and Deployment (Vertical scaling). +- **Safety**: Enforces guardrails (e.g., max scaling ratios, cooldown periods) to prevent instability. + +### 2. Mutating Webhook +Intercepts `Tortoise` CR creation and updates to: +- **Validate**: Ensure the configuration is valid (e.g., valid modes, resource policies). +- **Default**: Apply default values to optional fields (e.g., default `UpdateMode: Off`). + +## Internal Services + +The `TortoiseReconciler` delegates logic to several specialized services: + +- **`HpaService`**: Wrapper for interacting with HPA resources. Handles creation, updating, and synchronizing specs. +- **`VpaService`**: Wrapper for interacting with VPA resources. Manages the "Monitor-only" VPA that Tortoise uses to gather data. +- **`DeploymentService`**: Abstraction for Deployment operations. Handles fetching deployment details, scaling requests, and performing rolling restarts for vertical scaling. +- **`TortoiseService`**: Contains core domain logic for Tortoise. specific state transitions, finalizer management, and status updates. +- **`RecommenderService`**: The brain of the operation. It takes VPA recommendations and HPA status to calculate the optimal parameters (Min/Max Replicas, Target Utilization, Container Resource Requests). + +## Reconciliation Loop + +The `Reconcile` function in [`tortoise_controller.go`](./internal/controller/tortoise_controller.go) follows these steps: + +1. **Fetch & Validation**: + - Get the `Tortoise` CR. + - Handle deletion (remove HPA/VPA if `DeletionPolicy` allows, remove finalizers). + +2. **Resource Discovery**: + - Fetch the target Deployment. + - Fetch associated HPA and VPA. + +3. **Initialization** (Phase: `Initializing`): + - If HPA/VPA don't exist, create them. + - Set VPA to `Off` (Monitor) mode. + +4. **Policy Sync**: + - Ensure HPA spec matches the policy defined in Tortoise (e.g., sync metrics). + +5. **Data Gathering**: + - Check if VPA has produced recommendations. + - If not ready, wait (Phase: `GatheringData`). + +6. **Recommendation Calculation**: + - **Vertical**: detailed analysis of VPA's P90/Max recommendations vs current requests. + - **Horizontal**: Calculate optimal `MinReplicas` and `MaxReplicas` based on daily/weekly usage patterns. + - **Target Utilization**: specific formula to derive the best HPA target % based on resource headroom. + +7. **Actuation (Applying Changes)**: + - **HPA**: Update `Spec.MinReplicas`, `Spec.MaxReplicas`, and metrics (target utilization). + - **Vertical**: If `UpdateMode` is active and specific conditions are met (e.g., container needs resizing), direct update to Deployment's PodTemplate. + - *Note*: This triggers a Rolling Update. + +8. **Status Update**: + - Record recommendations and the current phase in `Tortoise.Status`. + +## State Machine (Tortoise Phases) + +Tortoise transitions through several phases during its lifecycle: + +| Phase | Description | +| :--- | :--- | +| `Initializing` | Initial state. Creating dependent resources (HPA, VPA). | +| `GatheringData` | Waiting for VPA to provide sufficient historical data. No actions taken yet. | +| `PartlyWorking` | Some recommendations are ready, but others (e.g., for a specific container) are still gathering data. | +| `Working` | Fully operational. Producing and applying recommendations. | +| `Emergency` | Emergency mode triggered (e.g., by user or critical metric failure). High resources/replicas enforced. | +| `BackToNormal` | Recovering from Emergency mode, gradually returning to normal operation. | + +## Data Flow Logic + +### From VPA to Optimization +Tortoise uses VPA strictly as a data source (Monitor Mode). +1. VPA calculates `Target` (P90) and `UncappedTarget` recommendations. +2. Tortoise reads these values. +3. **For Vertical Scaling**: + - Tortoise sets `request` = `VPA Recommendation` (with some smoothing/padding). +4. **For Horizontal Scaling**: + - Tortoise calculates `TargetUtilization` = `100 - (VPA_Max / Current_Request - Current_Target_Util)`. + - This complex formula ensures that HPA targets a utilization level that keeps the *actual* usage (P90) safely below 100%. + +### Safety Guardrails +To prevent thrashing or outages: +- **Vertical Scaling Frequency**: Limited to once per hour (by default) to avoid constant pod restarts. +- **Max Scale Down**: Prevents reducing resources too aggressively in a single step. From 36f835c81fdf282d943b4350f9b4857e6f711d15 Mon Sep 17 00:00:00 2001 From: Shreeharsh Shinde Date: Sat, 31 Jan 2026 16:49:39 +0530 Subject: [PATCH 2/4] feat: improve globalRecommendedBehaviour to scale up by pods Signed-off-by: Shreeharsh Shinde --- .../after/hpa.yaml | 40 +++-- .../after/deployment.yaml | 1 - .../after/hpa.yaml | 3 + .../after/tortoise.yaml | 62 +++---- .../after/hpa.yaml | 37 ++-- .../after/deployment.yaml | 12 +- .../after/hpa.yaml | 42 +++-- .../after/tortoise.yaml | 3 +- .../after/deployment.yaml | 12 +- .../after/hpa.yaml | 39 +++-- .../after/deployment.yaml | 12 +- .../after/hpa.yaml | 39 +++-- .../after/deployment.yaml | 15 +- .../after/hpa.yaml | 42 +++-- .../after/tortoise.yaml | 13 +- .../after/vpa-Monitor.yaml | 8 +- .../after/deployment.yaml | 12 +- .../after/hpa.yaml | 39 +++-- .../after/deployment.yaml | 16 +- .../after/hpa.yaml | 40 +++-- .../after/hpa.yaml | 40 +++-- .../after/hpa.yaml | 40 +++-- .../after/hpa.yaml | 29 ++-- .../after/hpa.yaml | 40 +++-- .../after/hpa.yaml | 40 +++-- .../after/hpa.yaml | 29 ++-- .../after/hpa.yaml | 26 +-- .../after/hpa.yaml | 29 ++-- .../after/hpa.yaml | 29 ++-- .../after/hpa.yaml | 29 ++-- .../after/hpa.yaml | 29 ++-- .../after/hpa.yaml | 66 ++++---- .../after/tortoise.yaml | 159 +++++++++--------- .../after/hpa.yaml | 29 ++-- .../after/hpa.yaml | 26 +-- .../after/hpa.yaml | 29 ++-- .../after/hpa.yaml | 29 ++-- .../after/hpa.yaml | 29 ++-- pkg/hpa/service.go | 5 + pkg/hpa/service_test.go | 5 + 40 files changed, 703 insertions(+), 521 deletions(-) diff --git a/internal/controller/testdata/mutable-autoscalingpolicy-add-another-horizontal/after/hpa.yaml b/internal/controller/testdata/mutable-autoscalingpolicy-add-another-horizontal/after/hpa.yaml index 7c627ee7..62ebce1e 100644 --- a/internal/controller/testdata/mutable-autoscalingpolicy-add-another-horizontal/after/hpa.yaml +++ b/internal/controller/testdata/mutable-autoscalingpolicy-add-another-horizontal/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 20 @@ -48,20 +51,25 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 - - containerResource: - container: istio-proxy - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + - containerResource: + container: istio-proxy + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/mutable-autoscalingpolicy-no-hpa-and-add-horizontal/after/deployment.yaml b/internal/controller/testdata/mutable-autoscalingpolicy-no-hpa-and-add-horizontal/after/deployment.yaml index 0d41a091..88d46d10 100644 --- a/internal/controller/testdata/mutable-autoscalingpolicy-no-hpa-and-add-horizontal/after/deployment.yaml +++ b/internal/controller/testdata/mutable-autoscalingpolicy-no-hpa-and-add-horizontal/after/deployment.yaml @@ -8,7 +8,6 @@ spec: strategy: {} template: metadata: - annotations: null creationTimestamp: null labels: app: mercari diff --git a/internal/controller/testdata/mutable-autoscalingpolicy-no-hpa-and-add-horizontal/after/hpa.yaml b/internal/controller/testdata/mutable-autoscalingpolicy-no-hpa-and-add-horizontal/after/hpa.yaml index 2d9ce3a5..0c5499f3 100644 --- a/internal/controller/testdata/mutable-autoscalingpolicy-no-hpa-and-add-horizontal/after/hpa.yaml +++ b/internal/controller/testdata/mutable-autoscalingpolicy-no-hpa-and-add-horizontal/after/hpa.yaml @@ -14,6 +14,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 10000 diff --git a/internal/controller/testdata/mutable-autoscalingpolicy-no-hpa-and-add-horizontal/after/tortoise.yaml b/internal/controller/testdata/mutable-autoscalingpolicy-no-hpa-and-add-horizontal/after/tortoise.yaml index 9af0d17e..b36841f1 100644 --- a/internal/controller/testdata/mutable-autoscalingpolicy-no-hpa-and-add-horizontal/after/tortoise.yaml +++ b/internal/controller/testdata/mutable-autoscalingpolicy-no-hpa-and-add-horizontal/after/tortoise.yaml @@ -20,20 +20,6 @@ status: cpu: Vertical memory: Vertical conditions: - containerResourceRequests: - - containerName: app - resource: - cpu: "10" - memory: 10Gi - - containerName: istio-proxy - resource: - cpu: "4" - memory: 4Gi - tortoiseConditions: - - lastTransitionTime: "2023-01-01T00:00:00Z" - lastUpdateTime: "2023-01-01T00:00:00Z" - status: "False" - type: FailedToReconcile containerRecommendationFromVPA: - containerName: app maxRecommendation: @@ -65,6 +51,37 @@ status: memory: quantity: 4Gi updatedAt: null + containerResourceRequests: + - containerName: app + resource: + cpu: "10" + memory: 10Gi + - containerName: istio-proxy + resource: + cpu: "4" + memory: 4Gi + tortoiseConditions: + - lastTransitionTime: "2023-01-01T00:00:00Z" + lastUpdateTime: "2023-01-01T00:00:00Z" + status: "False" + type: FailedToReconcile + containerResourcePhases: + - containerName: app + resourcePhases: + cpu: + lastTransitionTime: "2023-01-01T00:00:00Z" + phase: GatheringData + memory: + lastTransitionTime: "2023-01-01T00:00:00Z" + phase: Working + - containerName: istio-proxy + resourcePhases: + cpu: + lastTransitionTime: "2023-01-01T00:00:00Z" + phase: Working + memory: + lastTransitionTime: "2023-01-01T00:00:00Z" + phase: Working recommendations: horizontal: maxReplicas: @@ -94,23 +111,6 @@ status: cpu: "4" memory: 4Gi containerName: istio-proxy - containerResourcePhases: - - containerName: app - resourcePhases: - cpu: - lastTransitionTime: "2023-01-01T00:00:00Z" - phase: GatheringData - memory: - lastTransitionTime: "2023-01-01T00:00:00Z" - phase: Working - - containerName: istio-proxy - resourcePhases: - cpu: - lastTransitionTime: "2023-01-01T00:00:00Z" - phase: Working - memory: - lastTransitionTime: "2023-01-01T00:00:00Z" - phase: Working targets: horizontalPodAutoscaler: tortoise-hpa-mercari scaleTargetRef: diff --git a/internal/controller/testdata/mutable-autoscalingpolicy-remove-horizontal-2/after/hpa.yaml b/internal/controller/testdata/mutable-autoscalingpolicy-remove-horizontal-2/after/hpa.yaml index 28c7019f..8d14e91e 100644 --- a/internal/controller/testdata/mutable-autoscalingpolicy-remove-horizontal-2/after/hpa.yaml +++ b/internal/controller/testdata/mutable-autoscalingpolicy-remove-horizontal-2/after/hpa.yaml @@ -33,20 +33,25 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 - - containerResource: - container: istio-proxy - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + - containerResource: + container: istio-proxy + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-back-to-working/after/deployment.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-back-to-working/after/deployment.yaml index 96d10022..a3ec631c 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-back-to-working/after/deployment.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-back-to-working/after/deployment.yaml @@ -10,21 +10,21 @@ spec: metadata: annotations: kubectl.kubernetes.io/restartedAt: "2023-01-01T00:00:00Z" + sidecar.istio.io/inject: "true" + sidecar.istio.io/proxyCPU: "4" + sidecar.istio.io/proxyMemory: 4Gi creationTimestamp: null labels: app: mercari spec: containers: + - image: auto + name: istio-proxy + resources: {} - image: awesome-mercari-app-image name: app resources: requests: cpu: "10" memory: 10Gi - - image: awesome-istio-proxy-image - name: istio-proxy - resources: - requests: - cpu: "4" - memory: 4Gi status: {} diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-back-to-working/after/hpa.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-back-to-working/after/hpa.yaml index 3907eaea..1f6c9b78 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-back-to-working/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-back-to-working/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 20 @@ -41,22 +44,25 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 - type: ContainerResource - - containerResource: - container: istio-proxy - name: cpu - current: - value: 3 - type: ContainerResource \ No newline at end of file + - containerResource: + container: app + current: + value: "3" + name: cpu + type: ContainerResource + - containerResource: + container: istio-proxy + current: + value: "3" + name: cpu + type: ContainerResource + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-back-to-working/after/tortoise.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-back-to-working/after/tortoise.yaml index e36185be..fc5d7364 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-back-to-working/after/tortoise.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-back-to-working/after/tortoise.yaml @@ -70,7 +70,8 @@ status: type: HPATargetUtilizationUpdated - lastTransitionTime: "2023-01-01T00:00:00Z" lastUpdateTime: "2023-01-01T00:00:00Z" - message: the current number of replicas is not bigger than the preferred max replica number + message: the current number of replicas is not bigger than the preferred max + replica number reason: ScaledUpBasedOnPreferredMaxReplicas status: "False" type: ScaledUpBasedOnPreferredMaxReplicas diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-condition/after/deployment.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-condition/after/deployment.yaml index 96d10022..a3ec631c 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-condition/after/deployment.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-condition/after/deployment.yaml @@ -10,21 +10,21 @@ spec: metadata: annotations: kubectl.kubernetes.io/restartedAt: "2023-01-01T00:00:00Z" + sidecar.istio.io/inject: "true" + sidecar.istio.io/proxyCPU: "4" + sidecar.istio.io/proxyMemory: 4Gi creationTimestamp: null labels: app: mercari spec: containers: + - image: auto + name: istio-proxy + resources: {} - image: awesome-mercari-app-image name: app resources: requests: cpu: "10" memory: 10Gi - - image: awesome-istio-proxy-image - name: istio-proxy - resources: - requests: - cpu: "4" - memory: 4Gi status: {} diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-condition/after/hpa.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-condition/after/hpa.yaml index 9ec7057c..3b786a8a 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-condition/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-condition/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 15 @@ -41,21 +44,25 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "False" - type: ScalingActive - message: "the HPA was unable to compute the replica count: failed to get cpu utilization" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: 'the HPA was unable to compute the replica count: failed to get cpu utilization' + status: "False" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 0 - - containerResource: - container: istio-proxy - name: cpu - current: - value: 0 + - containerResource: + container: app + current: + value: "0" + name: cpu + type: "" + - containerResource: + container: istio-proxy + current: + value: "0" + name: cpu + type: "" desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-no-metrics/after/deployment.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-no-metrics/after/deployment.yaml index 96d10022..a3ec631c 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-no-metrics/after/deployment.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-no-metrics/after/deployment.yaml @@ -10,21 +10,21 @@ spec: metadata: annotations: kubectl.kubernetes.io/restartedAt: "2023-01-01T00:00:00Z" + sidecar.istio.io/inject: "true" + sidecar.istio.io/proxyCPU: "4" + sidecar.istio.io/proxyMemory: 4Gi creationTimestamp: null labels: app: mercari spec: containers: + - image: auto + name: istio-proxy + resources: {} - image: awesome-mercari-app-image name: app resources: requests: cpu: "10" memory: 10Gi - - image: awesome-istio-proxy-image - name: istio-proxy - resources: - requests: - cpu: "4" - memory: 4Gi status: {} diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-no-metrics/after/hpa.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-no-metrics/after/hpa.yaml index 45fa2a67..6cf55d2b 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-no-metrics/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-hpa-no-metrics/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 15 @@ -41,21 +44,25 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 0 - - containerResource: - container: istio-proxy - name: cpu - current: - value: 0 + - containerResource: + container: app + current: + value: "0" + name: cpu + type: "" + - containerResource: + container: istio-proxy + current: + value: "0" + name: cpu + type: "" desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/deployment.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/deployment.yaml index de7f9ffa..5d0a0696 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/deployment.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/deployment.yaml @@ -8,21 +8,22 @@ spec: strategy: {} template: metadata: + annotations: + sidecar.istio.io/inject: "true" + sidecar.istio.io/proxyCPU: "2" + sidecar.istio.io/proxyMemory: 2Gi creationTimestamp: null labels: app: mercari spec: containers: + - image: auto + name: istio-proxy + resources: {} - image: awesome-mercari-app-image name: app resources: requests: cpu: "6" - memory: 5Gi - - image: awesome-istio-proxy-image - name: istio-proxy - resources: - requests: - cpu: "2" - memory: 1.5Gi + memory: 6Gi status: {} diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/hpa.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/hpa.yaml index 13073ae1..9faa2843 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 8 @@ -41,22 +44,25 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 4 - type: ContainerResource - - containerResource: - container: istio-proxy - name: cpu - current: - value: 1 - type: ContainerResource \ No newline at end of file + - containerResource: + container: app + current: + value: "4" + name: cpu + type: ContainerResource + - containerResource: + container: istio-proxy + current: + value: "1" + name: cpu + type: ContainerResource + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/tortoise.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/tortoise.yaml index 420878d0..236ca692 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/tortoise.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/tortoise.yaml @@ -40,17 +40,17 @@ status: - containerName: istio-proxy maxRecommendation: cpu: - quantity: "1.5" + quantity: 1500m updatedAt: "2023-01-01T00:00:00Z" memory: - quantity: 1.5Gi + quantity: 1536Mi updatedAt: "2023-01-01T00:00:00Z" recommendation: cpu: - quantity: "1.5" + quantity: 1500m updatedAt: "2023-01-01T00:00:00Z" memory: - quantity: 1.5Gi + quantity: 1536Mi updatedAt: "2023-01-01T00:00:00Z" containerResourceRequests: - containerName: app @@ -79,7 +79,8 @@ status: type: FailedToReconcile - lastTransitionTime: "2023-01-01T00:00:00Z" lastUpdateTime: "2023-01-01T00:00:00Z" - message: the current number of replicas is not bigger than the preferred max replica number + message: the current number of replicas is not bigger than the preferred max + replica number reason: ScaledUpBasedOnPreferredMaxReplicas status: "False" type: ScaledUpBasedOnPreferredMaxReplicas @@ -129,7 +130,7 @@ status: containerName: app - RecommendedResource: cpu: "2" - memory: 1.5Gi + memory: 1536Mi containerName: istio-proxy targets: horizontalPodAutoscaler: tortoise-hpa-mercari diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/vpa-Monitor.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/vpa-Monitor.yaml index 4e5036b6..32ae79c0 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/vpa-Monitor.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-healthy/after/vpa-Monitor.yaml @@ -32,8 +32,8 @@ status: cpu: "1" memory: 1Gi target: - cpu: "1.5" - memory: 1.5Gi + cpu: 1500m + memory: 1536Mi upperBound: - cpu: "2.5" - memory: 2.5Gi + cpu: 2500m + memory: 2560Mi diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-unhealthy/after/deployment.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-unhealthy/after/deployment.yaml index 96d10022..a3ec631c 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-unhealthy/after/deployment.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-unhealthy/after/deployment.yaml @@ -10,21 +10,21 @@ spec: metadata: annotations: kubectl.kubernetes.io/restartedAt: "2023-01-01T00:00:00Z" + sidecar.istio.io/inject: "true" + sidecar.istio.io/proxyCPU: "4" + sidecar.istio.io/proxyMemory: 4Gi creationTimestamp: null labels: app: mercari spec: containers: + - image: auto + name: istio-proxy + resources: {} - image: awesome-mercari-app-image name: app resources: requests: cpu: "10" memory: 10Gi - - image: awesome-istio-proxy-image - name: istio-proxy - resources: - requests: - cpu: "4" - memory: 4Gi status: {} diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-unhealthy/after/hpa.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-unhealthy/after/hpa.yaml index 9ec7057c..3b786a8a 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-unhealthy/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-mixed-policies-unhealthy/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 15 @@ -41,21 +44,25 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "False" - type: ScalingActive - message: "the HPA was unable to compute the replica count: failed to get cpu utilization" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: 'the HPA was unable to compute the replica count: failed to get cpu utilization' + status: "False" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 0 - - containerResource: - container: istio-proxy - name: cpu - current: - value: 0 + - containerResource: + container: app + current: + value: "0" + name: cpu + type: "" + - containerResource: + container: istio-proxy + current: + value: "0" + name: cpu + type: "" desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-automatic-emergency-mode-vertical-scaling/after/deployment.yaml b/internal/controller/testdata/reconcile-automatic-emergency-mode-vertical-scaling/after/deployment.yaml index b2f5c2e9..a3ec631c 100644 --- a/internal/controller/testdata/reconcile-automatic-emergency-mode-vertical-scaling/after/deployment.yaml +++ b/internal/controller/testdata/reconcile-automatic-emergency-mode-vertical-scaling/after/deployment.yaml @@ -10,21 +10,21 @@ spec: metadata: annotations: kubectl.kubernetes.io/restartedAt: "2023-01-01T00:00:00Z" + sidecar.istio.io/inject: "true" + sidecar.istio.io/proxyCPU: "4" + sidecar.istio.io/proxyMemory: 4Gi creationTimestamp: null labels: app: mercari spec: containers: + - image: auto + name: istio-proxy + resources: {} - image: awesome-mercari-app-image name: app resources: requests: - cpu: "4" - memory: 6Gi - - image: awesome-istio-proxy-image - name: istio-proxy - resources: - requests: - cpu: "1" - memory: 1Gi + cpu: "10" + memory: 10Gi status: {} diff --git a/internal/controller/testdata/reconcile-for-the-istio-enabled-pod-working/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-istio-enabled-pod-working/after/hpa.yaml index 2d976100..e4aed4f9 100644 --- a/internal/controller/testdata/reconcile-for-the-istio-enabled-pod-working/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-istio-enabled-pod-working/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 20 @@ -41,20 +44,25 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 - - containerResource: - container: istio-proxy - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + - containerResource: + container: istio-proxy + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-during-emergency/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-during-emergency/after/hpa.yaml index a102d9c1..3fced02d 100644 --- a/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-during-emergency/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-during-emergency/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 15 @@ -41,20 +44,25 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 - - containerResource: - container: istio-proxy - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + - containerResource: + container: istio-proxy + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-emergency-started/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-emergency-started/after/hpa.yaml index a102d9c1..3fced02d 100644 --- a/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-emergency-started/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-emergency-started/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 15 @@ -41,20 +44,25 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 - - containerResource: - container: istio-proxy - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + - containerResource: + container: istio-proxy + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-one-off/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-one-off/after/hpa.yaml index bfc7d4d1..80688fae 100644 --- a/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-one-off/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-one-off/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 20 @@ -34,15 +37,19 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-suggested-too-small/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-suggested-too-small/after/hpa.yaml index 3dbd5407..ca9b908c 100644 --- a/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-suggested-too-small/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-suggested-too-small/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 20 @@ -41,20 +44,25 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 - - containerResource: - container: istio-proxy - name: cpu - current: - value: 3 \ No newline at end of file + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + - containerResource: + container: istio-proxy + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-working/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-working/after/hpa.yaml index 2d976100..e4aed4f9 100644 --- a/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-working/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-multiple-containers-pod-working/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 20 @@ -41,20 +44,25 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 - - containerResource: - container: istio-proxy - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + - containerResource: + container: istio-proxy + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-single-container-pod-backtonormal/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-single-container-pod-backtonormal/after/hpa.yaml index 87aabcd3..e02a86c7 100644 --- a/internal/controller/testdata/reconcile-for-the-single-container-pod-backtonormal/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-single-container-pod-backtonormal/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 20 @@ -34,15 +37,19 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-single-container-pod-dryrun/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-single-container-pod-dryrun/after/hpa.yaml index a0719772..ac9b1bae 100644 --- a/internal/controller/testdata/reconcile-for-the-single-container-pod-dryrun/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-single-container-pod-dryrun/after/hpa.yaml @@ -34,15 +34,19 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 \ No newline at end of file + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-single-container-pod-during-emergency/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-single-container-pod-during-emergency/after/hpa.yaml index b3fee935..2df29925 100644 --- a/internal/controller/testdata/reconcile-for-the-single-container-pod-during-emergency/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-single-container-pod-during-emergency/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 20 @@ -34,15 +37,19 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-single-container-pod-emergency-started/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-single-container-pod-emergency-started/after/hpa.yaml index b1b60f03..f23de02f 100644 --- a/internal/controller/testdata/reconcile-for-the-single-container-pod-emergency-started/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-single-container-pod-emergency-started/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 15 @@ -34,15 +37,19 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-single-container-pod-gathering-data-finished/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-single-container-pod-gathering-data-finished/after/hpa.yaml index 450a8cd5..2b4a9793 100644 --- a/internal/controller/testdata/reconcile-for-the-single-container-pod-gathering-data-finished/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-single-container-pod-gathering-data-finished/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 20 @@ -34,15 +37,19 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-single-container-pod-gathering-data/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-single-container-pod-gathering-data/after/hpa.yaml index a0c6bcf1..3c9fb5e7 100644 --- a/internal/controller/testdata/reconcile-for-the-single-container-pod-gathering-data/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-single-container-pod-gathering-data/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 100 @@ -34,15 +37,19 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-single-container-pod-hpa-changed-behavior/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-single-container-pod-hpa-changed-behavior/after/hpa.yaml index 3e015004..d9cf255d 100644 --- a/internal/controller/testdata/reconcile-for-the-single-container-pod-hpa-changed-behavior/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-single-container-pod-hpa-changed-behavior/after/hpa.yaml @@ -7,33 +7,33 @@ spec: behavior: scaleDown: policies: - - periodSeconds: 10 - type: Percent - value: 5 + - periodSeconds: 10 + type: Percent + value: 5 selectPolicy: Min scaleUp: policies: - - periodSeconds: 20 - type: Percent - value: 60 + - periodSeconds: 20 + type: Percent + value: 60 selectPolicy: Min stabilizationWindowSeconds: 50 maxReplicas: 100 metrics: - - external: - metric: - name: datadogmetric@hoge:hoge - target: - type: Value - value: "90" - type: External - - containerResource: - container: app - name: memory - target: - averageUtilization: 50 - type: Utilization - type: ContainerResource + - external: + metric: + name: datadogmetric@hoge:hoge + target: + type: Value + value: "90" + type: External + - containerResource: + container: app + name: memory + target: + averageUtilization: 50 + type: Utilization + type: ContainerResource minReplicas: 1 scaleTargetRef: apiVersion: apps/v1 @@ -41,15 +41,19 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-single-container-pod-hpa-changed-behavior/after/tortoise.yaml b/internal/controller/testdata/reconcile-for-the-single-container-pod-hpa-changed-behavior/after/tortoise.yaml index 27dd0bb9..039a2a0f 100644 --- a/internal/controller/testdata/reconcile-for-the-single-container-pod-hpa-changed-behavior/after/tortoise.yaml +++ b/internal/controller/testdata/reconcile-for-the-single-container-pod-hpa-changed-behavior/after/tortoise.yaml @@ -1,114 +1,113 @@ metadata: finalizers: - - tortoise.autoscaling.mercari.com/finalizer + - tortoise.autoscaling.mercari.com/finalizer name: mercari namespace: default spec: - targetRefs: - horizontalPodAutoscalerName: tortoise-hpa-mercari - scaleTargetRef: - apiVersion: apps/v1 - kind: Deployment - name: mercari-app horizontalPodAutoscalerBehavior: scaleDown: policies: - - periodSeconds: 10 - type: Percent - value: 5 + - periodSeconds: 10 + type: Percent + value: 5 selectPolicy: Min scaleUp: policies: - - periodSeconds: 20 - type: Percent - value: 60 + - periodSeconds: 20 + type: Percent + value: 60 selectPolicy: Min stabilizationWindowSeconds: 50 + targetRefs: + horizontalPodAutoscalerName: tortoise-hpa-mercari + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: mercari-app status: autoscalingPolicy: - - containerName: app - policy: - cpu: Vertical - memory: Horizontal + - containerName: app + policy: + cpu: Vertical + memory: Horizontal conditions: containerRecommendationFromVPA: - - containerName: app - maxRecommendation: - cpu: - quantity: "3" - updatedAt: "2023-01-01T00:00:00Z" - memory: - quantity: 3Gi - updatedAt: "2023-01-01T00:00:00Z" - recommendation: - cpu: - quantity: "3" - updatedAt: "2023-01-01T00:00:00Z" - memory: - quantity: 3Gi - updatedAt: "2023-01-01T00:00:00Z" - containerResourceRequests: - - containerName: app - resource: - cpu: "3" - memory: 4Gi - tortoiseConditions: - - lastTransitionTime: "2023-01-01T00:00:00Z" - lastUpdateTime: "2023-01-01T00:00:00Z" - message: - the current number of replicas is not bigger than the preferred max - replica number - reason: ScaledUpBasedOnPreferredMaxReplicas - status: "False" - type: ScaledUpBasedOnPreferredMaxReplicas - - lastTransitionTime: "2023-01-01T00:00:00Z" - lastUpdateTime: "2023-01-01T00:00:00Z" - message: The recommendation is provided - status: "True" - type: VerticalRecommendationUpdated - - lastTransitionTime: "2023-01-01T00:00:00Z" - lastUpdateTime: "2023-01-01T00:00:00Z" - status: "False" - type: FailedToReconcile - containerResourcePhases: - containerName: app - resourcePhases: + maxRecommendation: cpu: - lastTransitionTime: "2023-01-01T00:00:00Z" - phase: Working + quantity: "3" + updatedAt: "2023-01-01T00:00:00Z" memory: - lastTransitionTime: "2023-01-01T00:00:00Z" - phase: GatheringData + quantity: 3Gi + updatedAt: "2023-01-01T00:00:00Z" + recommendation: + cpu: + quantity: "3" + updatedAt: "2023-01-01T00:00:00Z" + memory: + quantity: 3Gi + updatedAt: "2023-01-01T00:00:00Z" + containerResourceRequests: + - containerName: app + resource: + cpu: "3" + memory: 4Gi + tortoiseConditions: + - lastTransitionTime: "2023-01-01T00:00:00Z" + lastUpdateTime: "2023-01-01T00:00:00Z" + message: the current number of replicas is not bigger than the preferred max + replica number + reason: ScaledUpBasedOnPreferredMaxReplicas + status: "False" + type: ScaledUpBasedOnPreferredMaxReplicas + - lastTransitionTime: "2023-01-01T00:00:00Z" + lastUpdateTime: "2023-01-01T00:00:00Z" + message: The recommendation is provided + status: "True" + type: VerticalRecommendationUpdated + - lastTransitionTime: "2023-01-01T00:00:00Z" + lastUpdateTime: "2023-01-01T00:00:00Z" + status: "False" + type: FailedToReconcile + containerResourcePhases: + - containerName: app + resourcePhases: + cpu: + lastTransitionTime: "2023-01-01T00:00:00Z" + phase: Working + memory: + lastTransitionTime: "2023-01-01T00:00:00Z" + phase: GatheringData recommendations: horizontal: maxReplicas: - - from: 0 - timezone: Local - to: 24 - updatedAt: "2023-01-01T00:00:00Z" - value: 20 + - from: 0 + timezone: Local + to: 24 + updatedAt: "2023-01-01T00:00:00Z" + value: 20 minReplicas: - - from: 0 - timezone: Local - to: 24 - updatedAt: "2023-01-01T00:00:00Z" - value: 5 + - from: 0 + timezone: Local + to: 24 + updatedAt: "2023-01-01T00:00:00Z" + value: 5 targetUtilizations: - - containerName: app - targetUtilization: - memory: 75 + - containerName: app + targetUtilization: + memory: 75 vertical: containerResourceRecommendation: - - RecommendedResource: - cpu: "3" - memory: 4Gi - containerName: app + - RecommendedResource: + cpu: "3" + memory: 4Gi + containerName: app targets: horizontalPodAutoscaler: tortoise-hpa-mercari scaleTargetRef: kind: "" name: "" verticalPodAutoscalers: - - name: tortoise-monitor-mercari - role: Monitor + - name: tortoise-monitor-mercari + role: Monitor tortoisePhase: PartlyWorking diff --git a/internal/controller/testdata/reconcile-for-the-single-container-pod-hpa-changed/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-single-container-pod-hpa-changed/after/hpa.yaml index cf3a7e97..ecc20d40 100644 --- a/internal/controller/testdata/reconcile-for-the-single-container-pod-hpa-changed/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-single-container-pod-hpa-changed/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 100 @@ -41,15 +44,19 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-single-container-pod-initializing/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-single-container-pod-initializing/after/hpa.yaml index a0c6bcf1..ac9b1bae 100644 --- a/internal/controller/testdata/reconcile-for-the-single-container-pod-initializing/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-single-container-pod-initializing/after/hpa.yaml @@ -34,15 +34,19 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-single-container-pod-partly-working/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-single-container-pod-partly-working/after/hpa.yaml index a0c6bcf1..3c9fb5e7 100644 --- a/internal/controller/testdata/reconcile-for-the-single-container-pod-partly-working/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-single-container-pod-partly-working/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 100 @@ -34,15 +37,19 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-single-container-pod-too-big/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-single-container-pod-too-big/after/hpa.yaml index 8ccddef9..bddba02c 100644 --- a/internal/controller/testdata/reconcile-for-the-single-container-pod-too-big/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-single-container-pod-too-big/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 10 @@ -41,15 +44,19 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/internal/controller/testdata/reconcile-for-the-single-container-pod-working/after/hpa.yaml b/internal/controller/testdata/reconcile-for-the-single-container-pod-working/after/hpa.yaml index 7c1e45ef..7ae098f2 100644 --- a/internal/controller/testdata/reconcile-for-the-single-container-pod-working/after/hpa.yaml +++ b/internal/controller/testdata/reconcile-for-the-single-container-pod-working/after/hpa.yaml @@ -16,6 +16,9 @@ spec: - periodSeconds: 60 type: Percent value: 100 + - periodSeconds: 60 + type: Pods + value: 4 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: 20 @@ -41,15 +44,19 @@ spec: name: mercari-app status: conditions: - - status: "True" - type: AbleToScale - message: "recommended size matches current size" - - status: "True" - type: ScalingActive - message: "the HPA was able to compute the replica count" + - lastTransitionTime: null + message: recommended size matches current size + status: "True" + type: AbleToScale + - lastTransitionTime: null + message: the HPA was able to compute the replica count + status: "True" + type: ScalingActive currentMetrics: - - containerResource: - container: app - name: cpu - current: - value: 3 + - containerResource: + container: app + current: + value: "3" + name: cpu + type: "" + desiredReplicas: 0 diff --git a/pkg/hpa/service.go b/pkg/hpa/service.go index 9886a7d7..89766055 100644 --- a/pkg/hpa/service.go +++ b/pkg/hpa/service.go @@ -55,6 +55,11 @@ var defaultHPABehaviorValue = &v2.HorizontalPodAutoscalerBehavior{ Value: 100, PeriodSeconds: 60, }, + { + Type: v2.PodsScalingPolicy, + Value: 4, + PeriodSeconds: 60, + }, }, }, ScaleDown: &v2.HPAScalingRules{ diff --git a/pkg/hpa/service_test.go b/pkg/hpa/service_test.go index 0c3e59b9..cf7d83f0 100644 --- a/pkg/hpa/service_test.go +++ b/pkg/hpa/service_test.go @@ -2905,6 +2905,11 @@ func TestService_InitializeHPA(t *testing.T) { Value: 100, PeriodSeconds: 60, }, + { + Type: v2.PodsScalingPolicy, + Value: 4, + PeriodSeconds: 60, + }, }, }, ScaleDown: &v2.HPAScalingRules{ From 6d45fb1cfce648e57b5d3368edc5db601807bc73 Mon Sep 17 00:00:00 2001 From: Shreeharsh Shinde <132091100+shreeharshshinde@users.noreply.github.com> Date: Sat, 31 Jan 2026 17:05:49 +0530 Subject: [PATCH 3/4] Delete docs/architecture.md --- docs/architecture.md | 128 ------------------------------------------- 1 file changed, 128 deletions(-) delete mode 100644 docs/architecture.md diff --git a/docs/architecture.md b/docs/architecture.md deleted file mode 100644 index 0cee2b80..00000000 --- a/docs/architecture.md +++ /dev/null @@ -1,128 +0,0 @@ -# Architecture - -This document provides a deep dive into the internal architecture of Tortoise. It is intended for contributors who want to understand how the controller works under the hood. - -## System Overview - -Tortoise creates a feedback loop between the user's workload (Deployment), the autoscalers (HPA, VPA), and the Tortoise optimization logic. - -```mermaid -graph TD - User[User] -->|Apply| Tortoise[Tortoise CR] - Tortoise -->|Reconcile| Controller[Tortoise Controller] - - subgraph "Managed Resources" - HPA[Horizontal Pod Autoscaler] - VPA[Vertical Pod Autoscaler] - end - - subgraph "Target Workload" - Deployment[Deployment] - ReplicaSet[ReplicaSet] - Pods[Pods] - end - - Controller -- 1. Creates/Updates --> HPA - Controller -- 2. Creates/Updates (Monitor Mode) --> VPA - - VPA -- 3. Analyzes Usage --> Pods - VPA -.->|Recommendation| Controller - - Controller -- 4. Calculates Optimization --> Recommendations[Internal Recommendations] - - Recommendations -- 5a. Updates Min/Max/Targets --> HPA - Recommendations -- 5b. Vertical Scaling (Rolling Update) --> Deployment - HPA -- 5c. Scales Replicas --> Deployment -``` - -## Core Components - -### 1. Tortoise Controller -The heart of the system. It runs a reconciliation loop that ensures the actual state matches the desired state defined in the `Tortoise` Custom Resource (CR). - -**Key Responsibilities:** -- **Orchestration**: Manages the lifecycle of associated HPA and VPA resources. -- **Optimization**: Calculates ideal resource requests and auto-scaling parameters based on historical data (from VPA) and current usage. -- **Actuation**: Applies changes to HPA (Horizontal scaling) and Deployment (Vertical scaling). -- **Safety**: Enforces guardrails (e.g., max scaling ratios, cooldown periods) to prevent instability. - -### 2. Mutating Webhook -Intercepts `Tortoise` CR creation and updates to: -- **Validate**: Ensure the configuration is valid (e.g., valid modes, resource policies). -- **Default**: Apply default values to optional fields (e.g., default `UpdateMode: Off`). - -## Internal Services - -The `TortoiseReconciler` delegates logic to several specialized services: - -- **`HpaService`**: Wrapper for interacting with HPA resources. Handles creation, updating, and synchronizing specs. -- **`VpaService`**: Wrapper for interacting with VPA resources. Manages the "Monitor-only" VPA that Tortoise uses to gather data. -- **`DeploymentService`**: Abstraction for Deployment operations. Handles fetching deployment details, scaling requests, and performing rolling restarts for vertical scaling. -- **`TortoiseService`**: Contains core domain logic for Tortoise. specific state transitions, finalizer management, and status updates. -- **`RecommenderService`**: The brain of the operation. It takes VPA recommendations and HPA status to calculate the optimal parameters (Min/Max Replicas, Target Utilization, Container Resource Requests). - -## Reconciliation Loop - -The `Reconcile` function in [`tortoise_controller.go`](./internal/controller/tortoise_controller.go) follows these steps: - -1. **Fetch & Validation**: - - Get the `Tortoise` CR. - - Handle deletion (remove HPA/VPA if `DeletionPolicy` allows, remove finalizers). - -2. **Resource Discovery**: - - Fetch the target Deployment. - - Fetch associated HPA and VPA. - -3. **Initialization** (Phase: `Initializing`): - - If HPA/VPA don't exist, create them. - - Set VPA to `Off` (Monitor) mode. - -4. **Policy Sync**: - - Ensure HPA spec matches the policy defined in Tortoise (e.g., sync metrics). - -5. **Data Gathering**: - - Check if VPA has produced recommendations. - - If not ready, wait (Phase: `GatheringData`). - -6. **Recommendation Calculation**: - - **Vertical**: detailed analysis of VPA's P90/Max recommendations vs current requests. - - **Horizontal**: Calculate optimal `MinReplicas` and `MaxReplicas` based on daily/weekly usage patterns. - - **Target Utilization**: specific formula to derive the best HPA target % based on resource headroom. - -7. **Actuation (Applying Changes)**: - - **HPA**: Update `Spec.MinReplicas`, `Spec.MaxReplicas`, and metrics (target utilization). - - **Vertical**: If `UpdateMode` is active and specific conditions are met (e.g., container needs resizing), direct update to Deployment's PodTemplate. - - *Note*: This triggers a Rolling Update. - -8. **Status Update**: - - Record recommendations and the current phase in `Tortoise.Status`. - -## State Machine (Tortoise Phases) - -Tortoise transitions through several phases during its lifecycle: - -| Phase | Description | -| :--- | :--- | -| `Initializing` | Initial state. Creating dependent resources (HPA, VPA). | -| `GatheringData` | Waiting for VPA to provide sufficient historical data. No actions taken yet. | -| `PartlyWorking` | Some recommendations are ready, but others (e.g., for a specific container) are still gathering data. | -| `Working` | Fully operational. Producing and applying recommendations. | -| `Emergency` | Emergency mode triggered (e.g., by user or critical metric failure). High resources/replicas enforced. | -| `BackToNormal` | Recovering from Emergency mode, gradually returning to normal operation. | - -## Data Flow Logic - -### From VPA to Optimization -Tortoise uses VPA strictly as a data source (Monitor Mode). -1. VPA calculates `Target` (P90) and `UncappedTarget` recommendations. -2. Tortoise reads these values. -3. **For Vertical Scaling**: - - Tortoise sets `request` = `VPA Recommendation` (with some smoothing/padding). -4. **For Horizontal Scaling**: - - Tortoise calculates `TargetUtilization` = `100 - (VPA_Max / Current_Request - Current_Target_Util)`. - - This complex formula ensures that HPA targets a utilization level that keeps the *actual* usage (P90) safely below 100%. - -### Safety Guardrails -To prevent thrashing or outages: -- **Vertical Scaling Frequency**: Limited to once per hour (by default) to avoid constant pod restarts. -- **Max Scale Down**: Prevents reducing resources too aggressively in a single step. From 4ee8db29f0c4002ff679574a20b1362f5a40c5b0 Mon Sep 17 00:00:00 2001 From: Shreeharsh Shinde <132091100+shreeharshshinde@users.noreply.github.com> Date: Tue, 3 Feb 2026 10:01:18 +0530 Subject: [PATCH 4/4] Remove architecture documentation link Removed architecture section from the README. --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index 7574b5c6..cf110a0c 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,6 @@ and how they can configure Tortoise so that they can let tortoises autoscale the - [Horizontal scaling](./docs/horizontal.md): describes how the Tortoise does the horizontal autoscaling internally. - [Vertical scaling](./docs/vertical.md): describes how the Tortoise does the vertical autoscaling internally. - [Technically details](./docs/internal.md): describes the technically details of Tortoise. (mostly for the contributors) -- [Architecture](./docs/architecture.md): describes the high-level architecture of Tortoise. - [Contributor guide](./docs/contributor-guide.md): describes other stuff for the contributor. (testing etc) ## API definition