Hello experts. I had a general question about flyte. I'm coming from authoring pytorchjobs as yaml files where the pytorchjobs just wrap a script. I see examples where the main logic of a task is defined in Python and wrapped with a decorator, and I presume it is injected into a pytorchjobs. Does flyte support just simply running a python/bash script in a pytorchjob (or any arbitrary custom resource) under the task decorator?
01/24/2024, 11:44 AM
hi Terry, i don't think that's possible today but it should be possible to use subprocess within a python task.
there's shell task but you cannot use pytorchjob with shell task as per this docstring.
But yes using shell tasks is an option. Remember using all gpus setting things up is more complicated
01/24/2024, 4:44 PM
Thanks for the response. Shell task looks nearly like what I'm looking for, with the exception of it not allowing me to specify an arbitrary job spec in the task_config. Any idea of when that option will be available?
As for elastic, that's helpful; tho I think that still requires the python code to be wrapped in the task decorator, and won't work with shell
01/24/2024, 4:49 PM
What do you mean by arbitrary job spec?
Do you want a task config to be passed into the shell task
01/24/2024, 6:17 PM
By arbitrary, I just meant like a pytorchjob or mpijob.
Re: shell task
Yea, since I'm trying to migrate users who wrap a shell script in a pytorchjob