Pryce
04/03/2023, 6:27 AMKetan (kumare3)
Dan Rammer (hamersaw)
04/03/2023, 2:04 PMPodTemplate
configuration for Flyte tasks through a few mechanisms (1) default PodTemplate
: basically create a k8s resource that gets applied on a per-namespace basis or (2) use the pod_template
or pod_template_name
arguments in the @task
decorator - it looks like docs are currently in PR, but they should outline this well.Pryce
04/03/2023, 6:36 PMDan Rammer (hamersaw)
04/03/2023, 6:38 PMPryce
04/03/2023, 6:48 PMDan Rammer (hamersaw)
04/03/2023, 8:38 PMPodTemplate
) you can create a k8s resource PodTemplate
that has the pvc and then this will be added to all k8s Pods that Flyte starts. It sounds like you want this to be only certain tasks. In this case the latter solution is probably what you're looking for. So the integration docs you linked are for the pod plugin, this functionality is actually a subset of newly added PodTemplate
support in the @task
decorators I mentioned. This is the issue that covered the work. Bascially you can add a pod_template
argument which specifies the PodTemplate
in the python code or create a k8s resource PodTemplate
which is applied at runtime. Using this paradigm you can add your pvc to individual tasks with something like:
@task(
request=Resources(),
pod_template=V1PodTemplate{
spec=V1PodSpec{
tolerations=[
V1Toleration{
key= "num-gpus"
operator= "Equal"
value= 1
effect= "NoSchedule"
},
]
}
}
)
The work on this pod_template
and pod_template_name
support is a superset of the pod plugin (integration docs you linked) because it can apply to more than just a python task. You can use pod_template
and pod_template_name
with ContainerTask
, ShellTask
, etc in addtiion to the KF operator plugins (pytorch, tensorflow, and MPI). Support for Ray, Dask, and Spark plugins isn't completed yet, but we're working on it.pod_template
or pod_template_name
in the @task
decorator is because some users would like to abstract these options (ie. complex k8s configuration) away from ML engineers. So the platform administrators can make a k8s resource PodTemplate
and then tell users to specify pod_template_name="template_a"
which injects the correct k8s configuration without the user having to understand k8s PodTemplate
.Pryce
04/03/2023, 9:27 PMflytectl register
as well to add them to the backend?Samhita Alla
Byron Hsu
04/05/2023, 5:42 AMpod_template_name
Dan Rammer (hamersaw)
04/05/2023, 3:27 PMkubectl create -f pod_template.yaml
where the pod_template.yaml file contains a valid k8s PodTemplate definition. For example, one of the ones I test with is:
apiVersion: v1
kind: PodTemplate
metadata:
name: flyte-template
namespace: flyte
template:
metadata:
labels:
pod-template-name: flyte-template
pod-template-namespace: flyte
spec:
containers:
- name: default
image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
imagePullPolicy: IfNotPresent
terminationMessagePath: "/dev/foo"
subdomain: "default-subdomain"
Pryce
04/05/2023, 10:08 PM