Hey all, I am trying to create dynamic PVCs and then assign them to my tasks (basically want to create a EBS volume to share during tasks in a workflow). I found some documentation here: https://github.com/flyteorg/flyte/issues/2754 kind of talking about it, but I cannot figure out how to actually assign this PVC to the task dynamically. I created a PVC using the kubernetes API with a random name (UUID), but am stuck on the task decorator and how to add this volume to the task:

@task(
    task_config=PodTemplate(
        pod_spec=V1PodSpec(
            containers=[
                client.V1Container(
                    name="main",
                    volume_mounts=[
                        client.V1VolumeMount(
                            name="shared-volume", mount_path="/mnt/shared-volume"
                        )
                    ]
                )
            ],
            volumes=[
                client.V1Volume(
                    name="shared-volume",
                    persistent_volume_claim=client.V1PersistentVolumeClaimVolumeSource(
                        claim_name=pvc_name, # The problem is here <---
                    ),
                )
            ]
        ),
    )
)
def consume_pvc_task(pvc_name: str) -> str:
    ...

Is there some way top access the pvc_name variable in the task decorator? Is what I am trying to do even possible or do I need to use kubernetes APIs to mount the volume and there is no native way in Flyte to do this dynamically? Thanks!

I am also curious if a way exists, so following this thread 👀

Ya, that is the issue I am having. You cannot define a PodTemplate dynamically at run time (at least from what I am seeing so far).

And then the real problem is that there is no dynamic mounting, right? The volumes/PVCs can be created and deleted in a setup/cleanup task calling the k8s api, but the mounting functionality is what is missing.

Correct, which is exactly what I have. My fallback is to actually mount the volume using the k8 api too...but ideally i would not have to

Yeah, that sounds dev unfriendly too much of a workaround

If I made one PodSpec which all of the tasks shared with a hardcoded volume, yes then they would all have the same PVC accessible in theory, but one task (which does not have the PVC mounted) needs to create the PVC first, for other tasks then to have it mounted after it's creation

I see. Hmm. I am guessing this behaviour (dynamic creation, mounting and deletion of PVCs) could be wrapped up in a plugin. i.e. this would be more or less similar to how Kubeflow does PVCs I guess: https://www.kubeflow.org/docs/components/pipelines/user-guides/core-functions/platform-specific-features/#kubernetes-persistentvolumeclaims

So I just spent a bunch of time trying to use

with_overrides

with a pod_template, which kind of works, but still not perfectly. I can pass a variable to this...but the variable can only be defined within the workflow, and not passed from a parent workflow. If it is passed from a parent_workflow, it breaks because the variable is suddenly a promise, and you cannot pass a promise to a PodSpec. So...still stuck at the moment with no way to dynamically create disk volumes and use them in Flyte natively (unless someone else has an idea on how to do this). Otherwise I have to probably do this via the kubernetes API directly instead and mount the volume within a task 😕

I guess one bit strange way to set it up is pre-create the PVC, have it set to RWX mode (read-write-many) and then use the same PVC everywhere? Maybe overprovision the claim itself and add file deletion/cleanup logic at the end of your workflows?

You can configure the PVC to automatically request/create the PV, so this is what I am doing. Where are you proposing to pre-create the PVC though, outside of Flyte? The main limiting factor is still the same of not being able to dynamically mount PV/PVCs to Flyte workflows.

Right. Would statically mounting the PVC be enough (and thus allowing you to use a predefined/existing one)? I'm guessing not otherwise you wouldn't ask

I do not believe so. The main problem I am trying to solve is have the ability to run my workflows for many customers in parallel, where each customer has one EBS volume we create and use in the workflow (shared by many tasks). By having a static name in the workflows of the PVC, we are limited by kubernetes in the sense that only one PVC can exist with one name in one namespace at the same time. So with this solution I can only process one customer at a time.

@glamorous-pizza-44029 at the beginning I thought the new support for overriding the Pod template at runtime would help you (https://github.com/flyteorg/flytekit/pull/2981) but I see that you probably need somewhat like a

.with_overrides

supporting a PVC as argument right?

Just to share my "POC" that I have been playing around with (so everyone is on the same page): https://gist.github.com/mastergberry/604bc795e9f6430b9c807a80c3feb29a You can see the problem on line #156, where I cannot register this workflow currently, as pvc_name is a promise, since it is coming from the

hoc_workflow

Could you make it into an eager workflow? https://docs.flyte.org/en/latest/user_guide/advanced_composition/eager_workflows.html#summary-of-workflows Those don't have promises, but come with other limitations so not sure if compatible for your case.

yeah the closest I can think to access outputs in this way is

@eager

I will play around with it and try it out. It seems to solve exactly the limitations we are having here. I just need to figure out how to set it up 😂 I see there are some credentials I need to generate to make it so this will work.

I think a plugin that handles the dynamic PVC lifecycle (create PVC per workflow execution, mount it on the all/selected tasks, delete once workflow execution reaches an end state (terminate, success, failure etc.)) might be a cleaner solution tho. Also would allow it to work with non-eager workflows

Which type of plugin are you reference? I see there are a few types.

I was thinking of a backend plugin https://docs.flyte.org/en/latest/user_guide/extending/backend_plugins.html Before PodTemplate became a first-class citizen in flytekit, you also needed a k8s pod plugin for this I believe. The only thing is that I see that plugins are meant for tasks instead of workflows, but I guess the plugin could provide 3 types of tasks (inspired by the kubeflow docs above): • CreatePVCTask: Probably the same as what you have in your PoC. This Task would be used to create a PVC and then return a name. • TaskWithMount: Just a regular old python task, but extended so you could give it a dynamic mount which the plugin would handle at runtime. • DeletePVCTask: Cleanup of the PVC and should be called only after all `TaskWithMount`s have ended running.

Hey! I just want to take a moment to thank you @gentle-tomato-480 for the detailed help 🙂 I finally figured out how to solve this, without actually needing a plugin or an eager workflow. It's a bit of a hackjob, but it works 🙂 I am sharing this just so you can see what I ended up doing: https://gist.github.com/mastergberry/604bc795e9f6430b9c807a80c3feb29a In the end, the solution was to use

@dynamic

with a few wrapper workflows. Workflow #1: Call Task to generate the Disk Name, call on_failure handler workflow Workflow #2: Simply used for on_error handler (since @dynamic does not support on_failure) Dynamic Workflow: Do the rest of the logic Since the dynamic workflow is not actually using promises (which became more clear from the eager documentation) and has real python variables, I can actually use

with_overrides

correctly 😄

Nice one!