Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hi. What do you think of this feature request? <https://github.com/flyteorg/flyte/issues/2883>

Hey <@UP4D9EY6T>! Great to hear from you, I actually saw this and have been thinking about it a bit. First off, I can see the benefit. I think this would be extremely powerful and other users are certainly interested. That being said, I think it is very important to implement something like this correctly, an unintuitive or otherwise problematic configuration will really make this difficult to understand and use. Have you thought about what this configuration may look like? If so, would you be able to elaborate? I would love to think about this a bit and then have a deeper discussion with anyone interested!

Thanks for looking into this! Yeah it is absolutely important to offer an intuitive config and make sure we don’t break backward compatibility. We have a plugin does this <https://github.com/spotify/flyte-flink-plugin/blob/072809cf4f594253d1a395e5887104433e4baa34/pkg/flink/auth.go#L25>, which is pretty much a copy of <https://github.com/flyteorg/flyteadmin/blob/997d2f76018e9ea8fc4cac5476a4de4c040cf0e4/pkg/runtime/interfaces/cluster_configuration.go#L10>

<@UP4D9EY6T> if I understand correctly then, this functionality is already baked into FlytePropeller (thanks!). So the only effort is porting these changes from the flink plugin you mentioned - we would probably want to commit the kubeclient creation into a flyteplugins package so it's uniform across plugins and could be used externally. Then configuration for specific plugins (ex. spark, ray, etc) could be done something like:
```plugins:
    k8s:
    spark:
        kube-client:
            name: "foo"
            endpoint: "<http://example.com|example.com>"
            enabled: true
    ray:
        kube-client:
            name: "bar"
            endpoint: "<http://example.com|example.com>"
            enabled: false```
Am I understanding this correctly?

I think internally we have discussed this at some length and in docs for this we're going to be very explicit that the perferred method to deploy Flyte is a single FlytePropeller instance per k8s cluster to a number of reasons ex. Pod ownership tieback, failure isolation, issues of cluster management and availability. but this is certainly a requested feature that we can support.

Yeah that’s pretty much it. We fully understand the preferred way is to have a propeller per cluster; and as you said this could be a reasonable feature that supports organizations with this need.