honnix

    honnix

    5 days ago
    Hi. What do you think of this feature request? https://github.com/flyteorg/flyte/issues/2883
    Dan Rammer (hamersaw)

    Dan Rammer (hamersaw)

    5 days ago
    Hey @honnix! Great to hear from you, I actually saw this and have been thinking about it a bit. First off, I can see the benefit. I think this would be extremely powerful and other users are certainly interested. That being said, I think it is very important to implement something like this correctly, an unintuitive or otherwise problematic configuration will really make this difficult to understand and use. Have you thought about what this configuration may look like? If so, would you be able to elaborate? I would love to think about this a bit and then have a deeper discussion with anyone interested!
    honnix

    honnix

    5 days ago
    Thanks for looking into this! Yeah it is absolutely important to offer an intuitive config and make sure we don’t break backward compatibility. We have a plugin does this https://github.com/spotify/flyte-flink-plugin/blob/072809cf4f594253d1a395e5887104433e4baa34/pkg/flink/auth.go#L25, which is pretty much a copy of https://github.com/flyteorg/flyteadmin/blob/997d2f76018e9ea8fc4cac5476a4de4c040cf0e4/pkg/runtime/interfaces/cluster_configuration.go#L10
    Dan Rammer (hamersaw)

    Dan Rammer (hamersaw)

    4 days ago
    @honnix if I understand correctly then, this functionality is already baked into FlytePropeller (thanks!). So the only effort is porting these changes from the flink plugin you mentioned - we would probably want to commit the kubeclient creation into a flyteplugins package so it's uniform across plugins and could be used externally. Then configuration for specific plugins (ex. spark, ray, etc) could be done something like:
    plugins:
        k8s:
        spark:
            kube-client:
                name: "foo"
                endpoint: "<http://example.com|example.com>"
                enabled: true
        ray:
            kube-client:
                name: "bar"
                endpoint: "<http://example.com|example.com>"
                enabled: false
    Am I understanding this correctly?
    I think internally we have discussed this at some length and in docs for this we're going to be very explicit that the perferred method to deploy Flyte is a single FlytePropeller instance per k8s cluster to a number of reasons ex. Pod ownership tieback, failure isolation, issues of cluster management and availability. but this is certainly a requested feature that we can support.
    honnix

    honnix

    4 days ago
    Yeah that’s pretty much it. We fully understand the preferred way is to have a propeller per cluster; and as you said this could be a reasonable feature that supports organizations with this need.