loud-belgium-4006
04/03/2023, 6:27 AMfreezing-airport-6809
freezing-airport-6809
hallowed-mouse-14616
04/03/2023, 2:04 PMhallowed-mouse-14616
04/03/2023, 2:06 PMPodTemplate
configuration for Flyte tasks through a few mechanisms (1) default PodTemplate
: basically create a k8s resource that gets applied on a per-namespace basis or (2) use the pod_template
or pod_template_name
arguments in the @task
decorator - it looks like docs are currently in PR, but they should outline this well.loud-belgium-4006
04/03/2023, 6:36 PMhallowed-mouse-14616
04/03/2023, 6:38 PMloud-belgium-4006
04/03/2023, 6:48 PMloud-belgium-4006
04/03/2023, 8:28 PMhallowed-mouse-14616
04/03/2023, 8:38 PMPodTemplate
) you can create a k8s resource PodTemplate
that has the pvc and then this will be added to all k8s Pods that Flyte starts. It sounds like you want this to be only certain tasks. In this case the latter solution is probably what you're looking for. So the integration docs you linked are for the pod plugin, this functionality is actually a subset of newly added PodTemplate
support in the @task
decorators I mentioned. This is the issue that covered the work. Bascially you can add a pod_template
argument which specifies the PodTemplate
in the python code or create a k8s resource PodTemplate
which is applied at runtime. Using this paradigm you can add your pvc to individual tasks with something like:
@task(
request=Resources(),
pod_template=V1PodTemplate{
spec=V1PodSpec{
tolerations=[
V1Toleration{
key= "num-gpus"
operator= "Equal"
value= 1
effect= "NoSchedule"
},
]
}
}
)
The work on this pod_template
and pod_template_name
support is a superset of the pod plugin (integration docs you linked) because it can apply to more than just a python task. You can use pod_template
and pod_template_name
with ContainerTask
, ShellTask
, etc in addtiion to the KF operator plugins (pytorch, tensorflow, and MPI). Support for Ray, Dask, and Spark plugins isn't completed yet, but we're working on it.hallowed-mouse-14616
04/03/2023, 8:41 PMpod_template
or pod_template_name
in the @task
decorator is because some users would like to abstract these options (ie. complex k8s configuration) away from ML engineers. So the platform administrators can make a k8s resource PodTemplate
and then tell users to specify pod_template_name="template_a"
which injects the correct k8s configuration without the user having to understand k8s PodTemplate
.loud-belgium-4006
04/03/2023, 9:27 PMflytectl register
as well to add them to the backend?tall-lock-23197
polite-ability-4005
04/05/2023, 5:42 AMpod_template_name
hallowed-mouse-14616
04/05/2023, 3:27 PMkubectl create -f pod_template.yaml
where the pod_template.yaml file contains a valid k8s PodTemplate definition. For example, one of the ones I test with is:
apiVersion: v1
kind: PodTemplate
metadata:
name: flyte-template
namespace: flyte
template:
metadata:
labels:
pod-template-name: flyte-template
pod-template-namespace: flyte
spec:
containers:
- name: default
image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
imagePullPolicy: IfNotPresent
terminationMessagePath: "/dev/foo"
subdomain: "default-subdomain"
loud-belgium-4006
04/05/2023, 10:08 PM