https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

anantharaman janakiraman

01/03/2024, 5:20 PM
Trying to run a dynamic workflow and getting this error when the dynamic workflow task is executed. We are trying to run the tasks on databricks via the plugin but the dynamic workflow step itself is executed on the k8s cluster. We are also passing the destination directory flag during pyflyte run. The error seems odd, any idea why this could be occurring?
Copy code
ValueError: Destination path is required to download distribution and it should 
be a directory
s

Samhita Alla

01/04/2024, 10:42 AM
is it a directory?
would you mind sharing the pyflyte run command?
a

anantharaman janakiraman

01/04/2024, 2:48 PM
Copy code
pyflyte run -p <project> -d development --remote --destination-dir "/databricks/driver" ./dynamic_wf/wf.py wf
s

Samhita Alla

01/04/2024, 4:02 PM
does that path exist in the image?
a

anantharaman janakiraman

01/04/2024, 4:05 PM
yes it does
s

Samhita Alla

01/04/2024, 4:07 PM
this is resulting in the error you're seeing: https://github.com/flyteorg/flytekit/blob/9bf7afae0f63510712d30e2e4fa5fbd219750b81/flytekit/tools/fast_registration.py#L107. for some reason, the isdir check is failing.
a

anantharaman janakiraman

01/04/2024, 4:12 PM
yeah I saw that. Checking the code now
s

Samhita Alla

01/04/2024, 4:13 PM
how are you building the image? are you using image spec?
a

anantharaman janakiraman

01/04/2024, 4:16 PM
no I am not using image spec. We are doing a conventional docker build using a dockerfile in our CI/CD workflow
s

Samhita Alla

01/04/2024, 4:18 PM
where are you specifying this image?
a

anantharaman janakiraman

01/04/2024, 4:22 PM
using this image in the dynamic workflow config like this
Copy code
@dynamic(container_image="<image_name>", cache=False, cache_version="1.0", limits=Resources(mem="2000Mi"))
def eval_model():
s

Samhita Alla

01/04/2024, 4:32 PM
i think specifying image in the pyflyte run command shouldn't result in the error you're seeing because then the code will be copied to the destination directory in the image you provide, not the default flytekit image.
can you give that a try?
a

anantharaman janakiraman

01/04/2024, 4:57 PM
sure let me try and confirm
thanks for your inputs!
2 Views