agreeable-flower-8989
08/01/2023, 3:41 PMloud-belgium-4006
08/01/2023, 5:53 PMagreeable-flower-8989
08/01/2023, 5:54 PMagreeable-flower-8989
08/01/2023, 5:56 PMFlyteFile
: the problem with this is for task code to access this config of FlyteFile, the code have to explicitly propagate this object to all the subsequent tasks right?loud-belgium-4006
08/01/2023, 5:56 PMagreeable-flower-8989
08/01/2023, 5:57 PM@task
def foo(arg1, arg2, config)
@task
def bar(arg1, arg2, config)
@workflow
def wf()
config = get_conf()
foo(..., config)
bar(..., config)
loud-belgium-4006
08/01/2023, 5:57 PMagreeable-flower-8989
08/01/2023, 5:58 PM@custom_workflow
def foo(your, input, arg, config_to_serialize):
your_normal_code(your, input, arg)
def custom_workflow(...)
# serialize the config object to S3
# and then dispatch the remaining code
@task
def your_normal_code(your, input, arg)
# note config object isn't explicitly passed as arg,
ec = access_config("executor_count")
loud-belgium-4006
08/01/2023, 5:59 PMloud-belgium-4006
08/01/2023, 5:59 PMagreeable-flower-8989
08/01/2023, 6:01 PMconfig_to_serializ
in my Naive solution 2agreeable-flower-8989
08/01/2023, 6:01 PMloud-belgium-4006
08/01/2023, 6:03 PMloud-belgium-4006
08/01/2023, 6:05 PMconfig.yaml
in the root of my workflow package and in the root __init__.py
I have:
import yaml
from pathlib import Path
ROOT_PATH = Path(__file__).resolve().parent
CONFIG_PATH = ROOT_PATH.joinpath('config.yaml')
with open(CONFIG_PATH, 'r') as f:
config = yaml.load(f, Loader=yaml.FullLoader)
The in all my modules I can access it with:
from run import config
@dynamic(
container_image=config['current_image'],
)
def process_samples(indir: FlyteDirectory, regs: List[str]) -> FlyteFile:
...
loud-belgium-4006
08/01/2023, 6:07 PMloud-belgium-4006
08/01/2023, 6:11 PMagreeable-flower-8989
08/01/2023, 11:00 PMIf you really needed to I suppose you could write a little convenience function that you can drop into tasks that need the functionality to pull a config from somewhereYah we do want config decoupled from the image.. (These config might specify how we want to configure resouce setting for some methods in children tasks, and we want to execute the same workflow with different configs)
agreeable-flower-8989
08/01/2023, 11:00 PMloud-belgium-4006
08/02/2023, 12:49 AM