<#4172 [Extended Resources] GPU Accelerators> Pull...
# flyte-github
a
#4172 [Extended Resources] GPU Accelerators Pull request opened by jeevb This PR introduces the concept of
ExtendedResources
that is meant to encapsulate all specialized resources that are not already captured by container resources (
v1.ResourceRequirements
), and adds GPU accelerator as one such resource. We plan on leveraging
ExtendedResources
for other specialized resources such as shared memory (
/dev/shm
) in the future. Implementation In this proposed implementation
ExtendedResources
is added as a field within
TaskTemplate
for task-level configuration, and
TaskNodeOverrides
for node-level overrides. We considered adding
ExtendedResources
directly to the IDL
Resources
object, but found that this is highly specific to container tasks, and not propagated correctly for pod templates and Pod tasks.
TaskTemplate
seemed like the most reasonable candidate outside of
Resources
. Generally, targeting tasks to specific GPU accelerators is a matter of setting the appropriate node affinities, and where relevant, tolerations. Mixed-partition multi-instance GPUs (MIGs) are an exception and have been excluded from the scope of this PR. We may add support for these in a future PR. When building a resource for a task requesting a specific GPU accelerator, FlytePropeller now injects: 1. Paired node selector requirement and toleration for the requested GPU device. The key is configurable via the
gpu-device-node-label
variable in the FlytePropeller k8s plugin configuration. A request for a T4 GPU might add the following to the resulting pod spec:
Copy code
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: cloud.google.com/gke-accelerator
            operator: In
            values:
            - nvidia-tesla-t4
...
  tolerations:
  - effect: NoSchedule
    key: cloud.google.com/gke-accelerator
    operator: Equal
    value: nvidia-tesla-t4
2. Paired node selector requirement and toleration for the requested GPU partition size (or unpartitioned). The key is configurable via the
gpu-partition-size-node-label
variable in the FlytePropeller k8s plugin configuration, and the node selector requirement and toleration to add for an unpartitioned MIG-capable GPU can be configured via
gpu-unpartitioned-node-selector-requirement
and
gpu-unpartitioned-toleration
respectively. A request for a partitioned or unpartitioned A100 GPU might add the following to the resulting pod spec:
Copy code
# 2g.10gb partition size
 spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: cloud.google.com/gke-accelerator
            operator: In
            values:
            - nvidia-tesla-a100
          - key: cloud.google.com/gke-gpu-partition-size
            operator: In
            values:
            - 2g.10gb
...
  tolerations:
  - effect: NoSchedule
    key: cloud.google.com/gke-accelerator
    operator: Equal
    value: nvidia-tesla-a100
  - effect: NoSchedule
    key: cloud.google.com/gke-gpu-partition-size
    operator: Equal
    value: 2g.10gb
 
 # Unparititioned
 spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: cloud.google.com/gke-accelerator
            operator: In
            values:
            - nvidia-tesla-a100
          - key: cloud.google.com/gke-gpu-partition-size
            operator: DoesNotExist
...
  tolerations:
  - effect: NoSchedule
    key: cloud.google.com/gke-accelerator
    operator: Equal
    value: nvidia-tesla-a100
  - effect: NoSchedule
    key: cloud.google.com/gke-gpu-partition-size
    operator: Equal
    value: DoesNotExist
Usage
ExtendedResources
are applied to resources generated by all K8s plugins, with the exception of Spark, for now. We believe that this change gives operators good control over targeting workloads to specific GPU-accelerated nodes just by tuning the labels and taints attached to those nodes. Consider a deployment with 3 different GPU-accelerated node groups: T4, 2g.10gb A100, unpartitioned A100. Operators may want to make the T4 node group the default, the 2g.10gb partitioned A100 node group the default A100 node group, and require that the unpartitioned A100 node group be explicitly requested for special workloads. This can be achieved as follows: 1. Create node groups with the following labels and taints: T4 node group
Copy code
labels:
    cloud.google.com/gke-accelerator: nvidia-tesla-t4  # auto
  taints:
    - key: nvidia.com/gpu  # auto
      value: present
      effect: "NO_SCHEDULE"
2g.10gb A100 node group
Copy code
labels:
    cloud.google.com/gke-accelerator: nvidia-tesla-a100  # auto
    cloud.google.com/gke-gpu-partition-size: 2g.10gb  # auto
  taints:
    - key: nvidia.com/gpu  # auto
      value: present
      effect: "NO_SCHEDULE"
    - key: cloud.google.com/gke-accelerator
      value: nvidia-tesla-a100
      effect: "NO_SCHEDULE"
Unpartitioned A100 node group
Copy code
labels:
    cloud.google.com/gke-accelerator: nvidia-tesla-a100  # auto
  taints:
    - key: nvidia.com/gpu  # auto
      value: present
      effect: "NO_SCHEDULE"
    - key: cloud.google.com/gke-accelerator
      value: nvidia-tesla-a100
      effect: "NO_SCHEDULE"
    - key: cloud.google.com/gke-gpu-partition-size
      value: DoesNotExist
      effect: "NO_SCHEDULE"
2. Add the following values to the FlytePropeller k8s plugin configuration:
Copy code
plugins:
  k8s:
    gpu-device-node-label: cloud.google.com/gke-accelerator
    gpu-partition-size-node-label: cloud.google.com/gke-gpu-partition-size
    gpu-unpartitioned-toleration:
      effect: NoSchedule
      key: cloud.google.com/gke-gpu-partition-size
      operator: Equal
      value: DoesNotExist
3. Users may now do one of the following: Request a GPU with no preference for device (Scheduled on T4 node group)
Copy code
@task(limits=Resources(gpu="1"))
def my_task() -> None:
    ...
with pod spec:
Copy code
apiVersion: v1
kind: Pod
spec:
  containers:
  - resources:
      limits:
        nvidia.com/gpu: 1
  tolerations:
  - key: nvidia.com/gpu  # auto
    operator: Equal
    value: present
    effect: NoSchedule
Explicitly request a T4 GPU
Copy code
@task(
    limits=Resources(gpu="1"),
    accelerator=T4,
)
def my_task() -> None:
    ...
with pod spec: ``` apiVersion: v1 kind: Pod spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-accelerator operator: In values: - nvidia-tesla-t4 containers: - resources: limits: <http://nvidia.com/gpu|n… flyteorg/flyte GitHub Actions: Build & Push Image GitHub Actions: Bump git tags ✅ 28 other checks have passed 28/30 successful checks