https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
c

Clint Zhang

10/12/2023, 4:42 PM
Hi team, we tried to launch a workflow using referenced launched plan. However, the execution only uses the uploaded resources files from the referenced workflow instead of the uploaded resources files in current workflow. Is there any workaround to overwrite the resources for referenced workflow?
cc: @Yee @Kevin Su @Ketan (kumare3)
j

Jay Ganbat

10/12/2023, 5:34 PM
Where are your resources defined? do you have any call signature to that workflow or examples
c

Clint Zhang

10/12/2023, 10:55 PM
The resources are defined in the current workflow
Found the alternative which is to use FlyteDirectory to download the resources from current workflow for remote workflow.
Wonder if we can exclude files/directory in FlyteDirectory?
y

Yee

10/12/2023, 11:13 PM
not entirely sure, could you elaborate? can you paste in a small snippet that shows what you’re trying to do?
c

Clint Zhang

10/12/2023, 11:16 PM
We have a task to upload the resources using FlyteDirectory:
Copy code
@task(
    container_image=IMAGE,
    limits=Resources(cpu="5", mem="10Gi"),
    requests=Resources(cpu="5", mem="10Gi"),
)
def upload_resource_folder() -> FlyteDirectory:
    return "./resources"
However some files/directories (like ./resources/build/*) under ./resources is not desired and is there a way to exclude ./resources/build in FlyteDirectory?
j

Jay Ganbat

10/12/2023, 11:20 PM
ohh so these files are baked into the image itself right, in that case its not possible to get that without either doing what you are doing or add the resource file into the reference workflows image right? Yee?
y

Yee

10/12/2023, 11:22 PM
no.. not with the current transformer it’s not possible. it will upload everything.
if you want to write your own though…
c

Clint Zhang

10/12/2023, 11:25 PM
I see. I can send a PR to add a exclude flag
y

Yee

10/12/2023, 11:25 PM
maybe
fast register respects git ignore
maybe that’s something to consider…
but that ties us to the git library even more and we’re trying to lighten the load on dependencies. in any case yeah, put in a pr. if it’s a new one, maybe put it under 
extras
and don’t register it by default. users who want it can register it. if it ends up being a tiny change that doesn’t change default behavior then we can just have one.
k

Ketan (kumare3)

10/12/2023, 11:37 PM
@Clint Zhang to ignore resources, i prefer using a streaming read/write system. Like using walk on the directory?
s

Samhita Alla

10/16/2023, 5:56 AM
y

Yee

10/16/2023, 6:43 AM
what did you mean btw when you said streaming read/write? like a transformer where the user provides a walk function?
Powered by