one query related to plugin development. flytekit and propeller plugins both can be developed using python itself? Or do we still need to use Python. Can someone guide us in plan development of plugin for k8s jobs. Currently flyte supports only pods.

what kind of k8s jobs you want to create? is that k8s batch job? currently, flyte can create a pod, tensorflow, torch, ray custom resource.

Hi, I want to deploy jobset which is custom resource. We need this to test our task where pods has to start at the same time. https://jobset.sigs.k8s.io/docs/

there are two styles of plugins yes. there’s the backend plugins, written in go, and compiled into the flyte backend engine. you can read more about that here: https://docs.flyte.org/en/latest/user_guide/extending/backend_plugins.html#backend-plugins these have to be written in go. they can’t be written in python.

For pods to start at same time you can simply use a scheduler right? But jobset does look interesting. To create one - this is a kubernetes special plugin the go side provides a simpler api - 3 methods Pay attention to the docs that yee sent, this section https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.26/go/tasks/pluginmachinery/k8s#Plugin

Hi Ketan, thank you for the info :). Pods run on different nodes independently and with some constraints. So we need a mechanism to start them all at once or nothing. Seems jobsets suits there.

Ohh you can use a gang scheduler

will it support different pod spec? means three pods has to start together with different spec for each