Hi all! I'm gearing up to move over to flyte and am curious about the best way to approach a rather niche requirement. Certain tasks of mine currently rely on a daemon running in my k8s cluster to execute. The tasks are k8s jobs that connect to this daemon via a url (exposed using k8s service) and a unix socket (exposed via a shared pvc). What would be the best way to make this daemon available to flyte tasks? My naive approach from reading through the docs is to manually deploy the daemon and then modify the tasks pod via flytekitplugins.pod; is there a better way? Thank you!

Pod templates

cc @Eduardo Apolinario (eapolinario) docs finished on this yet?

Awesome, thank you so much! I'll keep an eye on that PR.

Just to be very clear, all of this support it already merged in Flyte 1.4. Just documentation is pending - so you can use these features right now!

Gotcha - I probably need the docs to spoon feed me tbh 👀

Correct, so for examples. In the case of the former (ie. default

PodTemplate

) you can create a k8s resource

PodTemplate

that has the pvc and then this will be added to all k8s Pods that Flyte starts. It sounds like you want this to be only certain tasks. In this case the latter solution is probably what you're looking for. So the integration docs you linked are for the pod plugin, this functionality is actually a subset of newly added

PodTemplate

support in the

@task

decorators I mentioned. This is the issue that covered the work. Bascially you can add a

pod_template

argument which specifies the

PodTemplate

in the python code or create a k8s resource

PodTemplate

which is applied at runtime. Using this paradigm you can add your pvc to individual tasks with something like:

@task(
    request=Resources(),
    pod_template=V1PodTemplate{
        spec=V1PodSpec{
            tolerations=[
                V1Toleration{
                    key= "num-gpus"
                    operator= "Equal"
                    value= 1
                    effect= "NoSchedule"
                },
            ]
        }
    }
)

The work on this

pod_template

and

pod_template_name

support is a superset of the pod plugin (integration docs you linked) because it can apply to more than just a python task. You can use

pod_template

and

pod_template_name

with

ContainerTask

,

ShellTask

, etc in addtiion to the KF operator plugins (pytorch, tensorflow, and MPI). Support for Ray, Dask, and Spark plugins isn't completed yet, but we're working on it.

Ahh awesome okay, makes a lot of sense. Yeah I like pod_template_name a lot, keeps the deco clean. I'm guessing those pod templates will use something like

flytectl register

as well to add them to the backend?

I don't think so. After you create a k8s pod template, it automatically gets picked up and applied when you specify its name. Should we have more docs on how one should go about this process — creation of pod template, etc.? @Dan Rammer (hamersaw)

Yes @Samhita Alla is correct. If you need to deploy the podtemplate manually to the backend if you want to use

pod_template_name

@Pryce you can add this with something like

kubectl create -f pod_template.yaml

where the pod_template.yaml file contains a valid k8s PodTemplate definition. For example, one of the ones I test with is:

apiVersion: v1
kind: PodTemplate
metadata:
  name: flyte-template
  namespace: flyte
template:
  metadata:
    labels:
      pod-template-name: flyte-template
      pod-template-namespace: flyte
  spec:
    containers:
      - name: default
        image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
        imagePullPolicy: IfNotPresent
        terminationMessagePath: "/dev/foo"
    subdomain: "default-subdomain"

Oh I see, you create it explicitly via kubectl in the flyte namespace and then you can reference it from your task definitions. Thanks for clarifying!