Hey team, Flyte 2 looks really exciting! We've been using Flyte for a few years now and are currently evaluating Flyte 2. Among other projects, we might want to use it to launch tasks on a Slurm cluster. I saw that there was some support for this in Flyte 1, although this is not linked in the documentation at the moment. Could you give us a quick assessment on how well Flyte can integrate with Slurm in general and how this could look like for Flyte 2 specifically? Would Flyte 2 be a good option or would an alternative that already integrates with Slurm be preferred in this case? Thanks!

We have been looking at slurm and evaluating the available approaches.

@happy-crayon-35655 a couple of things that may be of interest to you • recent blog post one of our engineers wrote on SLURM union.ai/blog-post/a-modern-alternative-to-slurm-for-ml-workloads • union.ai/docs/…/agent-plugins#…

@happy-crayon-35655 i would also like to talk to you and understand the usecases for slurm. We have been cooking something that I believe can challenge slurm 🤞 but would love to work closer with you and learn if you are interested

@modern-parrot-90373 We want to set up an ML pipeline on their infra (and have been using Flyte for similar projects in the past), but they can only provision compute resources via Slurm, so we thought we can keep doing the orchestration in Flyte and launch the jobs on Slurm only

do the teams like the interactive nature of SLURM?

Not sure what their experience has been; Slurm is used as a part of a larger organization there so we'll not get around using it, unfortunately. Regarding the initial question, do you see a viable way to use Flyte for launching jobs on the Slurm cluster or would it be advisable to use an orchestrator which integrates with it already, such as Nextflow?

Slurm connector does launch jobs in slurm - you just have to port it to v2