Mick Jermsurawong
08/01/2023, 3:41 PMPryce
08/01/2023, 5:53 PMMick Jermsurawong
08/01/2023, 5:54 PMFlyteFile
: the problem with this is for task code to access this config of FlyteFile, the code have to explicitly propagate this object to all the subsequent tasks right?Pryce
08/01/2023, 5:56 PMMick Jermsurawong
08/01/2023, 5:57 PM@task
def foo(arg1, arg2, config)
@task
def bar(arg1, arg2, config)
@workflow
def wf()
config = get_conf()
foo(..., config)
bar(..., config)
Pryce
08/01/2023, 5:57 PMMick Jermsurawong
08/01/2023, 5:58 PM@custom_workflow
def foo(your, input, arg, config_to_serialize):
your_normal_code(your, input, arg)
def custom_workflow(...)
# serialize the config object to S3
# and then dispatch the remaining code
@task
def your_normal_code(your, input, arg)
# note config object isn't explicitly passed as arg,
ec = access_config("executor_count")
Pryce
08/01/2023, 5:59 PMMick Jermsurawong
08/01/2023, 6:01 PMconfig_to_serializ
in my Naive solution 2Pryce
08/01/2023, 6:03 PMconfig.yaml
in the root of my workflow package and in the root __init__.py
I have:
import yaml
from pathlib import Path
ROOT_PATH = Path(__file__).resolve().parent
CONFIG_PATH = ROOT_PATH.joinpath('config.yaml')
with open(CONFIG_PATH, 'r') as f:
config = yaml.load(f, Loader=yaml.FullLoader)
The in all my modules I can access it with:
from run import config
@dynamic(
container_image=config['current_image'],
)
def process_samples(indir: FlyteDirectory, regs: List[str]) -> FlyteFile:
...
Mick Jermsurawong
08/01/2023, 11:00 PMIf you really needed to I suppose you could write a little convenience function that you can drop into tasks that need the functionality to pull a config from somewhereYah we do want config decoupled from the image.. (These config might specify how we want to configure resouce setting for some methods in children tasks, and we want to execute the same workflow with different configs)
Pryce
08/02/2023, 12:49 AM