Tom Catshoek
09/13/2023, 3:17 PMif __name__ == "__main__":
with Pool(4) as p:
models = p.map(training_workflow, [{"C": 0.1}, {"C": 0.2}, {"C": 0.3}, {"C": 0.4}])
for the workflow in the getting started tutorial https://docs.flyte.org/projects/cookbook/en/latest/index.html
This does not work because apparently the workflows don't play nice with pickling
_pickle.PicklingError: Can't pickle <function training_workflow at 0x7fccd751eb60>: it's not the same object as __main__.training_workflow
I would like to do this because I'm writing a CLI tool that does some data manipulation on a bunch of files using flexible pipelines and flyte seems like a very nice way of defining these, with the type checking between tasks etc. Eventually these pipelines should also run remotely, but for now my focus is still on running things locally. I would also like to avoid requiring the users of the tool to spin up a demo flyte cluster.
Maybe this is a bad idea and flyte is the wrong tool for this job, but if that's the case feel free to tell me also.Jay Ganbat
09/13/2023, 4:37 PMwait
on execution it wont hang.
So you can just submit all your work in a loop and then monitor the execution object at the endTom Catshoek
09/13/2023, 6:05 PMJay Ganbat
09/13/2023, 7:37 PMTom Catshoek
09/13/2023, 7:38 PMValueError: TaskFunction cannot be a nested/inner or local function. It should
be accessible at a module level for Flyte to execute it. Test modules with names
beginning with `test_` are allowed to have nested tasks. If you're decorating
your task function with custom decorators, use functools.wraps or
functools.update_wrapper on the function wrapper. Alternatively if you want to
create your own tasks with custom behavior use the TaskResolverMixin
Jay Ganbat
09/13/2023, 7:40 PM._workflow_function
Tom Catshoek
09/13/2023, 7:50 PMJay Ganbat
09/13/2023, 7:52 PMTom Catshoek
09/13/2023, 7:52 PMKetan (kumare3)
Jay Ganbat
09/13/2023, 9:08 PMKetan (kumare3)