Detailed answers 1. (For 1). you will have to write a backend-plugin that can schedule onto a slurm cluster. this will use the Slurm WebAPI. You can probably use the Slurm REST API. This can use the WebAPI backend plugin system to write this using a simple CRUD interface. Checkout examples of Databricks, Snowflake etc. We are in the process of updating this to make this even better and easier to author - even in python 2. Yes, Open source Flyte can load balance, but it needs ability to communicate with the cluster from the control plane (flyteAdmin) K8s API server and Flytepropeller (in the dataplane) should be able to send grpc messages back to the control plane. Refer to Union.ai for a more seamless setup 3. Absolutely. We call this extensible data persistence layer. Flyte is compatible with any S3 like blob store. The API for this is extensible in 2 places. Before that understand how Flyte handles data - using this doc a. Extend the backend to allow a new metastore like S3. this is done using flytestdlib storage api. This API already uses the stow system underneath so if it is already extended in stow, then it should work out of box. b. python data persistence will need access to metastore and raw store. This can be extended using protocol based system and depends on library API like - fsspec. Hopefully this answers

Btw @Giacomo Dabisias IT how’s it going?

We currently do not have a slurm backend plugin

Thanks!

Can you talk about your requirements

I want a system that will keep track of code and data in a reproducible manner, and have a nice way to embed visual summaries (plots, tables) in a nice Web GUI front-end.

I guess the key requirements are: • be able to submit jobs to the Slurm cluster • track code and loaded packages used for each job • track data on both the local parallel file system and S3 buckets ◦ ideally it would also manage data retention policies, but that is a later concern • provide a web view of currents status, logs and visual summaries

Console outputs and debugging failed jobs should also be requirements I think.

(I admit I'm not that familiar with Flyte, and my Kubernetes knowledge is pretty basic)

Slurm is a batch scheduler, so the job will be in a queue until the requested resources are available

If I submit a job with SLURM will the cluster already be running with a large number of CPUs/GPUs? Or it can autoscale as per the job.

Yes, we can typically assume it is already running (powering up/down nodes is handled by the Slurm controller)

Also @Simon Byrne backend extensions already exist in golang as described here https://docs.flyte.org/projects/cookbook/en/latest/auto/core/extend_flyte/backend_plugins.html