Skip to content

Multizone clusters with GPU_NODE_MIN_SIZE=1 must have a GPU quota of at least 2. #367

Description

@willgraf

Describe the bug
Deploying a cluster in multiple zones requires a GPU quota equal to the number of zones if GPU_NODE_MIN_SIZE is at least 1. If the GPU quota is only 1, a multizone cluster will fail with the error:

ERROR: (gcloud.container.node-pools.create) ResponseError: code=403, message=Insufficient regional quota to satisfy request: resource "PREEMPTIBLE_NVIDIA_V100_GPUS": request requires '2.0' and is short '1.0'. project has a quota of '1.0' with '1.0' available. View and manage quotas at https://console.cloud.google.com/iam-admin/quotas?usage=USED.

To Reproduce
Steps to reproduce the behavior:

  1. During cluster configuration, select a GPU type with a quota of 1.
  2. Select a minimum GPU node count of at least 1.
  3. Select a multizone cluster configuration.
  4. Create the cluster.
  5. See the error in the failed cluster creation output.

Expected behavior
Ideally, the cluster will not allow the user to start creating a cluster with a bad configuration.

Metadata

Metadata

Assignees

No one assigned

    Labels

    wontfixThis will not be worked on

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions