https://flyte.org
Join Slack
Hi team, we tried to launch a workflow using referenced launched plan. However, the execution only u...
# flyte-support
b
Hi team, we tried to launch a workflow using referenced launched plan. However, the execution only uses the uploaded resources files from the referenced workflow instead of the uploaded resources files in current workflow. Is there any workaround to overwrite the resources for referenced workflow?
cc: @thankful-minister-83577 @glamorous-carpet-83516 @freezing-airport-6809
m
Where are your resources defined? do you have any call signature to that workflow or examples
b
The resources are defined in the current workflow
Found the alternative which is to use FlyteDirectory to download the resources from current workflow for remote workflow.
Wonder if we can exclude files/directory in FlyteDirectory?
t
not entirely sure, could you elaborate? can you paste in a small snippet that shows what you’re trying to do?
b
We have a task to upload the resources using FlyteDirectory:
Copy code
@task(
    container_image=IMAGE,
    limits=Resources(cpu="5", mem="10Gi"),
    requests=Resources(cpu="5", mem="10Gi"),
)
def upload_resource_folder() -> FlyteDirectory:
    return "./resources"
However some files/directories (like ./resources/build/*) under ./resources is not desired and is there a way to exclude ./resources/build in FlyteDirectory?
m
ohh so these files are baked into the image itself right, in that case its not possible to get that without either doing what you are doing or add the resource file into the reference workflows image right? Yee?
t
no.. not with the current transformer it’s not possible. it will upload everything.
if you want to write your own though…
b
I see. I can send a PR to add a exclude flag
t
maybe
fast register respects git ignore
maybe that’s something to consider…
but that ties us to the git library even more and we’re trying to lighten the load on dependencies. in any case yeah, put in a pr. if it’s a new one, maybe put it under 
extras
and don’t register it by default. users who want it can register it. if it ends up being a tiny change that doesn’t change default behavior then we can just have one.
f
@brief-boots-17198 to ignore resources, i prefer using a streaming read/write system. Like using walk on the directory?
t
t
what did you mean btw when you said streaming read/write? like a transformer where the user provides a walk function?
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by