little-nightfall-98272
02/28/2024, 7:50 PMmake
• Marking certain nodes so their commands gets exclusive access to a machine (i.e. no parallel execution)
• Dynamically working with files, such as when a step generates a number of files that's known only after it is finished
Thank you.freezing-airport-6809
pyflyte run wf.py wf
should run everything locally. Or simply execute the python function
2. All tasks are executed in their isolated containers, thats like having exclusive access to machines. Dont worry about parallel executions, as each of them will use different containers. ofcourse you can serialize things too
3. yes
Example of files
from flytekit import FlyteFile, FlyteDirector
@task
def foo() -> Flytefile:
...
@task
def foo2(f: FlyteFile) -> FlyteDirectory:
...
@workflow
def wf() -> FlyteDirectory:
f = foo()
return foo2(f=f)
If you see we are simply passing files in between. This guarantees only one works on the file and dont worry about pollution flyte will take care of making copies etclittle-nightfall-98272
02/29/2024, 6:52 PMfreezing-airport-6809