Hi team, we tried to launch a workflow using refer...
# ask-the-communityc
Clint Zhang
10/12/2023, 4:42 PMHi team, we tried to launch a workflow using referenced launched plan. However, the execution only uses the uploaded resources files from the referenced workflow instead of the uploaded resources files in current workflow. Is there any workaround to overwrite the resources for referenced workflow?
Clint Zhang
10/12/2023, 4:55 PMcc: @Yee @Kevin Su @Ketan (kumare3)
j
Jay Ganbat
10/12/2023, 5:34 PMWhere are your resources defined? do you have any call signature to that workflow or examples
c
Clint Zhang
10/12/2023, 10:55 PMThe resources are defined in the current workflow
Clint Zhang
10/12/2023, 10:55 PMFound the alternative which is to use FlyteDirectory to download the resources from current workflow for remote workflow.
Clint Zhang
10/12/2023, 10:56 PMWonder if we can exclude files/directory in FlyteDirectory?
y
Yee
10/12/2023, 11:13 PM
not entirely sure, could you elaborate? can you paste in a small snippet that shows what you’re trying to do?
c
Clint Zhang
10/12/2023, 11:16 PMWe have a task to upload the resources using FlyteDirectory:
Copy code
@task(
container_image=IMAGE,
limits=Resources(cpu="5", mem="10Gi"),
requests=Resources(cpu="5", mem="10Gi"),
)
def upload_resource_folder() -> FlyteDirectory:
return "./resources"j
Jay Ganbat
10/12/2023, 11:20 PMohh so these files are baked into the image itself right, in that case its not possible to get that without either doing what you are doing or add the resource file into the reference workflows image
right? Yee?
y
Yee
10/12/2023, 11:22 PM
no.. not with the current transformer it’s not possible. it will upload everything.
Yee
10/12/2023, 11:24 PM
Yee
10/12/2023, 11:25 PM
if you want to write your own though…
c
Clint Zhang
10/12/2023, 11:25 PMI see. I can send a PR to add a exclude flag
y
Yee
10/12/2023, 11:25 PM
maybe
Yee
10/12/2023, 11:26 PM
fast register respects git ignore
Yee
10/12/2023, 11:26 PM
maybe that’s something to consider…
Yee
10/12/2023, 11:30 PM
but that ties us to the git library even more and we’re trying to lighten the load on dependencies. in any case yeah, put in a pr. if it’s a new one, maybe put it under
extrask
Ketan (kumare3)
10/12/2023, 11:37 PM
@Clint Zhang to ignore resources, i prefer using a streaming read/write system. Like using walk on the directory?
s
Samhita Alla
10/16/2023, 5:56 AM
Here's more about streaming: https://flyte.org/blog/flyte-1-5-streaming-support-pod-templates-partial-tasks-and-more#stream-files-and-directories
y
Yee
10/16/2023, 6:43 AM
what did you mean btw when you said streaming read/write? like a transformer where the user provides a walk function?