acoustic-carpenter-78188
10/06/2023, 3:31 AMExtendedResources that is meant to encapsulate all specialized resources that are not already captured by container resources (v1.ResourceRequirements), and adds GPU accelerator as one such resource. We plan on leveraging ExtendedResources for other specialized resources such as shared memory (/dev/shm) in the future.
Implementation
In this proposed implementation ExtendedResources is added as a field within TaskTemplate for task-level configuration, and TaskNodeOverrides for node-level overrides. We considered adding ExtendedResources directly to the IDL Resources object, but found that this is highly specific to container tasks, and not propagated correctly for pod templates and Pod tasks. TaskTemplate seemed like the most reasonable candidate outside of Resources.
Generally, targeting tasks to specific GPU accelerators is a matter of setting the appropriate node affinities, and where relevant, tolerations. Mixed-partition multi-instance GPUs (MIGs) are an exception and have been excluded from the scope of this PR. We may add support for these in a future PR.
When building a resource for a task requesting a specific GPU accelerator, FlytePropeller now injects:
1. Paired node selector requirement and toleration for the requested GPU device. The key is configurable via the gpu-device-node-label variable in the FlytePropeller k8s plugin configuration. A request for a T4 GPU might add the following to the resulting pod spec:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.google.com/gke-accelerator
operator: In
values:
- nvidia-tesla-t4
...
tolerations:
- effect: NoSchedule
key: cloud.google.com/gke-accelerator
operator: Equal
value: nvidia-tesla-t4
2. Paired node selector requirement and toleration for the requested GPU partition size (or unpartitioned). The key is configurable via the gpu-partition-size-node-label variable in the FlytePropeller k8s plugin configuration, and the node selector requirement and toleration to add for an unpartitioned MIG-capable GPU can be configured via gpu-unpartitioned-node-selector-requirement and gpu-unpartitioned-toleration respectively. A request for a partitioned or unpartitioned A100 GPU might add the following to the resulting pod spec:
# 2g.10gb partition size
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.google.com/gke-accelerator
operator: In
values:
- nvidia-tesla-a100
- key: cloud.google.com/gke-gpu-partition-size
operator: In
values:
- 2g.10gb
...
tolerations:
- effect: NoSchedule
key: cloud.google.com/gke-accelerator
operator: Equal
value: nvidia-tesla-a100
- effect: NoSchedule
key: cloud.google.com/gke-gpu-partition-size
operator: Equal
value: 2g.10gb
# Unparititioned
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.google.com/gke-accelerator
operator: In
values:
- nvidia-tesla-a100
- key: cloud.google.com/gke-gpu-partition-size
operator: DoesNotExist
...
tolerations:
- effect: NoSchedule
key: cloud.google.com/gke-accelerator
operator: Equal
value: nvidia-tesla-a100
- effect: NoSchedule
key: cloud.google.com/gke-gpu-partition-size
operator: Equal
value: DoesNotExist
Usage
ExtendedResources are applied to resources generated by all K8s plugins, with the exception of Spark, for now. We believe that this change gives operators good control over targeting workloads to specific GPU-accelerated nodes just by tuning the labels and taints attached to those nodes. Consider a deployment with 3 different GPU-accelerated node groups: T4, 2g.10gb A100, unpartitioned A100. Operators may want to make the T4 node group the default, the 2g.10gb partitioned A100 node group the default A100 node group, and require that the unpartitioned A100 node group be explicitly requested for special workloads. This can be achieved as follows:
1. Create node groups with the following labels and taints:
T4 node group
labels:
cloud.google.com/gke-accelerator: nvidia-tesla-t4 # auto
taints:
- key: nvidia.com/gpu # auto
value: present
effect: "NO_SCHEDULE"
2g.10gb A100 node group
labels:
cloud.google.com/gke-accelerator: nvidia-tesla-a100 # auto
cloud.google.com/gke-gpu-partition-size: 2g.10gb # auto
taints:
- key: nvidia.com/gpu # auto
value: present
effect: "NO_SCHEDULE"
- key: cloud.google.com/gke-accelerator
value: nvidia-tesla-a100
effect: "NO_SCHEDULE"
Unpartitioned A100 node group
labels:
cloud.google.com/gke-accelerator: nvidia-tesla-a100 # auto
taints:
- key: nvidia.com/gpu # auto
value: present
effect: "NO_SCHEDULE"
- key: cloud.google.com/gke-accelerator
value: nvidia-tesla-a100
effect: "NO_SCHEDULE"
- key: cloud.google.com/gke-gpu-partition-size
value: DoesNotExist
effect: "NO_SCHEDULE"
2. Add the following values to the FlytePropeller k8s plugin configuration:
plugins:
k8s:
gpu-device-node-label: cloud.google.com/gke-accelerator
gpu-partition-size-node-label: cloud.google.com/gke-gpu-partition-size
gpu-unpartitioned-toleration:
effect: NoSchedule
key: cloud.google.com/gke-gpu-partition-size
operator: Equal
value: DoesNotExist
3. Users may now do one of the following:
Request a GPU with no preference for device (Scheduled on T4 node group)
@task(limits=Resources(gpu="1"))
def my_task() -> None:
...
with pod spec:
apiVersion: v1
kind: Pod
spec:
containers:
- resources:
limits:
nvidia.com/gpu: 1
tolerations:
- key: nvidia.com/gpu # auto
operator: Equal
value: present
effect: NoSchedule
Explicitly request a T4 GPU
@task(
limits=Resources(gpu="1"),
accelerator=T4,
)
def my_task() -> None:
...
with pod spec:
```
apiVersion: v1
kind: Pod
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.google.com/gke-accelerator
operator: In
values:
- nvidia-tesla-t4
containers:
- resources:
limits:
<http://nvidia.com/gpu|n…
flyteorg/flyte
GitHub Actions: Build & Push Image
GitHub Actions: Bump git tags
✅ 28 other checks have passed
28/30 successful checksacoustic-carpenter-78188
10/06/2023, 3:31 AMacoustic-carpenter-78188
10/12/2023, 4:42 PM