Trying to run a dynamic workflow and getting this error when the dynamic workflow task is executed. We are trying to run the tasks on databricks via the plugin but the dynamic workflow step itself is executed on the k8s cluster. We are also passing the destination directory flag during pyflyte run. The error seems odd, any idea why this could be occurring?

ValueError: Destination path is required to download distribution and it should 
be a directory

is it a directory?

does that path exist in the image?

yes it does

this is resulting in the error you're seeing: https://github.com/flyteorg/flytekit/blob/9bf7afae0f63510712d30e2e4fa5fbd219750b81/flytekit/tools/fast_registration.py#L107. for some reason, the isdir check is failing.

yeah I saw that. Checking the code now

how are you building the image? are you using image spec?

no I am not using image spec. We are doing a conventional docker build using a dockerfile in our CI/CD workflow

where are you specifying this image?

using this image in the dynamic workflow config like this

@dynamic(container_image="<image_name>", cache=False, cache_version="1.0", limits=Resources(mem="2000Mi"))
def eval_model():

i think specifying image in the pyflyte run command shouldn't result in the error you're seeing because then the code will be copied to the destination directory in the image you provide, not the default flytekit image.

sure let me try and confirm