[Feature] RayService Incremental Upgrade Project Tracker

[ryanaoleary]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

The following list defines the scope of the alpha for the RayService Incremental Upgrade feature.

• Implement Ray Serve APIs for users to define custom timeouts for readiness
  • per this discussion this may not be necessary, we can revisit whether additional timeouts from the KubeRay side for a stuck upgrade are necessary after the initial implementation
• RayService Incremental Upgrade Alpha
  • Before merging, we should ensure the following:
    • Test and document rollback behavior. Full rollback support for the cases defined in the REP can be implemented in a follow-up PR.
    • API review for the new RayService fields added
    • Load/stress testing to ensure no requests are dropped
• Add tests for upgrade rollback/roll-forward cases
  • A->B->A case. This is where the RayService is upgraded from current state A to goal state B. In the middle of the upgrade, the goal state is changed back to A. Alternatively, a controller error during the upgrade should also result in us rolling back to state A. The expected behavior is that the RayService will rollback to the original state.
  • A->B->C case. This is where the RayService is upgraded from current state A to goal state B. In the middle of the upgrade, the goal state is changed to state C. The expected behavior is that the RayService will rollback to state A, and then upgrade again from A to C.

Use case

It'd be useful to support an API to incrementally upgrade the RayService, scaling a new RayCluster to handle only a % capacity of the total traffic to the RayService in order to avoid delays in the upgrade process due to resource constraints. This issue will track the progress of implementing the RayService incremental upgrade feature.

Related issues

REP: https://github.com/ray-project/enhancements/blob/main/reps/2024-12-4-ray-service-incr-upgrade.md

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[ryanaoleary]

cc: @MortalHappiness

[ryanaoleary]

cc: @Future-Outlier This tracking issue is a pretty rough outline of the alpha scope, I can continue to add details so that it's more helpful. If anything sticks out to you, feel free to add to/comment on this issue.

[Future-Outlier]

TODO: We should document the resources we will use to teach users how to decide the parameters.
The total resource will be as follows:

Formula

R_total_worker_pods = ceil(R_base_worker × (1 + MaxSurgePercent/100))
R_total(t) = R_head_A + R_head_B + R_total_worker_pods × R_per_worker

Example

Configuration

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: example-cluster
spec:
  headGroupSpec:
    template:
      spec:
        containers:
        - name: ray-head
          resources:
            requests:
              cpu: "1"
              memory: "2Gi"
  workerGroupSpecs:
  - replicas: 2
    template:
      spec:
        containers:
        - name: ray-worker
          resources:
            requests:
              cpu: "2"
              memory: "4Gi"

Resource Calculation

Head Pod Resources:

R_head_A = 1 CPU + 2Gi memory
R_head_B = 1 CPU + 2Gi memory
R_head_total = 2 CPU + 4Gi memory

Worker Pod Base Resources:

R_base_worker_cpu = 2 replicas × 2 CPU = 4 CPU cores
R_base_worker_memory = 2 replicas × 4Gi = 8Gi memory

Incremental Upgrade Total Resource Formula:

R_total_worker_pods = ceil(R_base_worker_count × (1 + MaxSurgePercent/100))
R_total_cpu(t) = R_head_total_cpu + R_total_worker_pods × R_per_worker_cpu
R_total_memory(t) = R_head_total_memory + R_total_worker_pods × R_per_worker_memory

Actual Calculation (MaxSurgePercent = 40%)

Step 1: Calculate Total Worker Pods

R_base_worker_count = 2
MaxSurgePercent = 40%
R_total_worker_pods = ceil(2 × (1 + 40/100)) = ceil(2 × 1.4) = ceil(2.8) = 3

Step 2: Calculate Total Resources

R_per_worker_cpu = 2 CPU
R_per_worker_memory = 4Gi memory

R_total_worker_cpu = 3 × 2 = 6 CPU cores
R_total_worker_memory = 3 × 4Gi = 12Gi memory

R_total_cpu(t) = 2 + 6 = 8 CPU cores
R_total_memory(t) = 4Gi + 12Gi = 16Gi memory

Step 3: Final Result

R_total(t) = 8 CPU cores + 16Gi memory

[Future-Outlier]

Hi, @ryanaoleary
do you mind add a doc about this feature?
I will help review this week?

[ryanaoleary]

Hi, @ryanaoleary do you mind add a doc about this feature? I will help review this week?

Sure sounds good, I'll start writing an API overview + user guide to add to the Ray docs