Hello, I am just evaluating different workflow managing tools for performing experiments, and I would like to ask if Flyte can do the following:
• Running locally like
make
• Marking certain nodes so their commands gets exclusive access to a machine (i.e. no parallel execution)
• Dynamically working with files, such as when a step generates a number of files that's known only after it is finished
Thank you.
f
freezing-airport-6809
02/28/2024, 11:04 PM
@little-nightfall-98272 let me answer your question
1.
pyflyte run wf.py wf
should run everything locally. Or simply execute the python function
2. All tasks are executed in their isolated containers, thats like having exclusive access to machines. Dont worry about parallel executions, as each of them will use different containers. ofcourse you can serialize things too
3. yes
Example of files
If you see we are simply passing files in between. This guarantees only one works on the file and dont worry about pollution flyte will take care of making copies etc
l
little-nightfall-98272
02/29/2024, 6:52 PM
For 2, I don't think that's what I mean. Some of my task involve benchmarking the physical machine it runs on, so these tasks actually require exclusive access to a physical machine, not a container. Is this possible?
f
freezing-airport-6809
03/01/2024, 4:41 AM
Yup this is possible using pod templates and setting right node labels and selectors in k8s