Hi all! I'm gearing up to move over to flyte and a...
# ask-the-community
p
Hi all! I'm gearing up to move over to flyte and am curious about the best way to approach a rather niche requirement. Certain tasks of mine currently rely on a daemon running in my k8s cluster to execute. The tasks are k8s jobs that connect to this daemon via a url (exposed using k8s service) and a unix socket (exposed via a shared pvc). What would be the best way to make this daemon available to flyte tasks? My naive approach from reading through the docs is to manually deploy the daemon and then modify the tasks pod via flytekitplugins.pod; is there a better way? Thank you!
k
Pod templates
Let me ask few folks to share docs - @Byron Hsu / @Dan Rammer (hamersaw)
d
cc @Eduardo Apolinario (eapolinario) docs finished on this yet?
@Pryce we expose
PodTemplate
configuration for Flyte tasks through a few mechanisms (1) default
PodTemplate
: basically create a k8s resource that gets applied on a per-namespace basis or (2) use the
pod_template
or
pod_template_name
arguments in the
@task
decorator - it looks like docs are currently in PR, but they should outline this well.
p
Awesome, thank you so much! I'll keep an eye on that PR.
d
Just to be very clear, all of this support it already merged in Flyte 1.4. Just documentation is pending - so you can use these features right now!
p
Gotcha - I probably need the docs to spoon feed me tbh 👀
@Dan Rammer (hamersaw) just to solidify my understanding.. am i correct that the deployment docs around k8s specify a default pod template on the backend whereas the integration docs allow you to specify a per-task pod template? So for my specific usecase, I would deploy my daemon to the cluster, unbeknownst to flyte, and then modify my pod template in one of the above ways to communicate with it?
d
Correct, so for examples. In the case of the former (ie. default
PodTemplate
) you can create a k8s resource
PodTemplate
that has the pvc and then this will be added to all k8s Pods that Flyte starts. It sounds like you want this to be only certain tasks. In this case the latter solution is probably what you're looking for. So the integration docs you linked are for the pod plugin, this functionality is actually a subset of newly added
PodTemplate
support in the
@task
decorators I mentioned. This is the issue that covered the work. Bascially you can add a
pod_template
argument which specifies the
PodTemplate
in the python code or create a k8s resource
PodTemplate
which is applied at runtime. Using this paradigm you can add your pvc to individual tasks with something like:
Copy code
@task(
    request=Resources(),
    pod_template=V1PodTemplate{
        spec=V1PodSpec{
            tolerations=[
                V1Toleration{
                    key= "num-gpus"
                    operator= "Equal"
                    value= 1
                    effect= "NoSchedule"
                },
            ]
        }
    }
)
The work on this
pod_template
and
pod_template_name
support is a superset of the pod plugin (integration docs you linked) because it can apply to more than just a python task. You can use
pod_template
and
pod_template_name
with
ContainerTask
,
ShellTask
, etc in addtiion to the KF operator plugins (pytorch, tensorflow, and MPI). Support for Ray, Dask, and Spark plugins isn't completed yet, but we're working on it.
the distinction between using either
pod_template
or
pod_template_name
in the
@task
decorator is because some users would like to abstract these options (ie. complex k8s configuration) away from ML engineers. So the platform administrators can make a k8s resource
PodTemplate
and then tell users to specify
pod_template_name="template_a"
which injects the correct k8s configuration without the user having to understand k8s
PodTemplate
.
p
Ahh awesome okay, makes a lot of sense. Yeah I like pod_template_name a lot, keeps the deco clean. I'm guessing those pod templates will use something like
flytectl register
as well to add them to the backend?
s
I don't think so. After you create a k8s pod template, it automatically gets picked up and applied when you specify its name. Should we have more docs on how one should go about this process — creation of pod template, etc.? @Dan Rammer (hamersaw)
b
Yes @Samhita Alla is correct. If you need to deploy the podtemplate manually to the backend if you want to use
pod_template_name
d
@Pryce you can add this with something like
kubectl create -f pod_template.yaml
where the pod_template.yaml file contains a valid k8s PodTemplate definition. For example, one of the ones I test with is:
Copy code
apiVersion: v1
kind: PodTemplate
metadata:
  name: flyte-template
  namespace: flyte
template:
  metadata:
    labels:
      pod-template-name: flyte-template
      pod-template-namespace: flyte
  spec:
    containers:
      - name: default
        image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
        imagePullPolicy: IfNotPresent
        terminationMessagePath: "/dev/foo"
    subdomain: "default-subdomain"
p
Oh I see, you create it explicitly via kubectl in the flyte namespace and then you can reference it from your task definitions. Thanks for clarifying!
107 Views