https://flyte.org
Join Slack
Hi, is it possible to mount a persistent volume into a container task in flyte? or to specify a PodT...
# flyte-support
s
Hi, is it possible to mount a persistent volume into a container task in flyte? or to specify a PodTemplate? i see run-time and compile-time options but they don't really mesh with the container task syntax
l
Currently all tasks use the default pod template that has been defined.. we would be open to either define a new template and use that or have some parameter to mount the PVC. We came across some historical threads but they weren’t for container tasks, hence the confusion.
f
hmm what do you mean? you can define pod templates
l
yes.. I found it later. It was a bit hard to find in the docs after digging through a bunch of old conversations and lots of searching :)
f
@little-cricket-84530 mind contributing
l
apologize for the slightly dumb question.. but is flyte version == flytekit version?
Happy to help with documentation…
f
Versions are on release but flight backend is backwards compatible
l
I mean does flyte version mean flytekit version? I’m trying to go through our helm charts and I’m a bit lost
So I defined a persistent volume claim… created a pod template which references this PVC (can share code snippets if required) and used it to create a ContainerTask. I don’t think the pod is getting provisioned at all.. Flyte UI simply shows it in “Queued” state
Copy code
pt = PodTemplate(
        pod_spec=V1PodSpec(
            containers=[
                V1Container(
                    name="my-container",
                    volume_mounts=[
                        V1VolumeMount(
                            name="my-volume",
                            mount_path="/scratch"
                        )
                    ]
                )
            ],
            volumes=[
                V1Volume(
                    name="my-volume",
                    persistent_volume_claim=V1PersistentVolumeClaimVolumeSource(
                            claim_name="flyte-pod-pvc"
                        )
                    )
            ]
        )

container_task = ContainerTask(
        name="my-task",
        input_data_dir="/var/inputs",
        output_data_dir="/var/outputs",
        inputs=kwtypes(script=str),
        requests=Resources(ephemeral_storage="90Gi", mem="100Gi", cpu="32"),
        image=f"xxxx:{docker_tag}",
        command=[
            "bash",
            "-e",
            "/var/inputs/script"
        ],
        pod_template=pt
    )
In my yaml files I have 
flyte-pod-pvc
defined
Copy code
RuntimeExecutionError: max number of system retry attempts [31/30] exhausted. Last known status message: failed at Node[dn0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: container not specified in task template
@freezing-airport-6809 any idea what might be up? Or can you point me to anyone or any docs/resources?
s
Copy code
No plugin found for Handler-type [raw-container], defaulting to [container]
f
hmm that is ok, you do not have a container specified
your task is wrong, not sure how you built it
l
task is wrong as in?
for some history.. the task works without the template.. but runs out of memory. Hence trying to attach a PVC to it
f
will take a look later - sorry busy at the moment
👍🏼 1
s
could it be that our flyte version predates container tasks, hence why raw-container plugin is missing?
f
nope, raw containers have existed for a while, its just that we migrated everything to have pod templates. So it is possible that you are using an older verison of flyte backend that does not support pod templates
47 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by