Would it be possible to expose a

NodeSelector

field in the

ExecutionSpec

in flyteidl/admin? Labels and annotations are already exposed

sure, would you mind creating a PR for this?

I don’t think we want to add this

hi @Ketan (kumare3) our use case is to pass dynamic values -- during execution time -- in this nodeSelector.. PodTemplate doesn't allow us to specify that. Podtemplate is static.

How does the user know about the nodes? This will lead to non reproducible and weird behavior. My suggestion is to use pod template names in the back

Our use case • In organization that requires some workload segmentation, there is necessary contract between user and compute infra. Eg. Pods of

teamA

will have nodeSelector

<http://company.com/team|company.com/team>: teamA

and the node should have a corresponding label. ◦ This is actually Karpenter standard pattern "to provide workload segregation across nodes", where provisioner will create new nodes with label corresponding to pending pods value of the

NodeSelector

• We apply nodeSelector at execution time, and expect the scheduling behavior to be per execution. Essentially, we are ok with different registered workflow/task having different nodeSelector. (Unlike pod template name) • We have admin-proxy backend, where we have a reliable user identity and we would like to inject NodeSelector into the Execution request spec (similarly to how labels/annotation customizable on Flyte UI is applied to all tasks) Using pod template names • i know Dan works with the community for this useful feature, but i think it'll be hard for us.. ◦ we want nodeSelector per user, so we will need to create that many number of podTemplates when new users in the company get onboarded.. ◦ Pod template name is per-task, and we want to apply this scheduling for all tasks in a workflow. ◦ More importantly, users can choose any podTemplate meant for other users, so for our use case to achieve security objectives on user-specific workload isolation, during runtime, i can specify pod template for nodeSelector of any other users

I don't think adding node selectors at that layer is what you want. Not all plugins allow flyte to customize the pods, fully, that end up getting created.. I think what you really want is a single pod template (default) that says "nodeselector: pod.labels.user_identity" Assuming flyte passes down the user principal as a trusted label.. 1. I think it's reasonable to add that label in flyteadmin (if you haven't already done that in your proxy) 2. K8s doesn't allow downstream APIs (referencing pod labels) in node selectors I don't think.. but this can be achieved through an admission webhook (flyte's or kyverno or others)

Not all plugins allow flyte to customize the pods, fully, that end up getting created..

1/ i see. Would "labels" added as part of execution request also get propagated to all plugins? But what you are saying is rather, the nodeSelector is less general than labels such that the selector might be dropped in some cases?

pod template (default) that says "nodeselector: pod.labels.user_identity"

...

K8s doesn't allow downstream APIs (referencing pod labels) in node selectors

2/ Just to confirm, you are saying

nodeselector: pod.labels.user_identity

as in idea right? Implementation-wise, you are suggesting the admission webhook.

admission webhook (flyte's )

3/ Are there ways we can add on to Flyte's admission webhook?

1/ exactly... 2/ you got it! 3/ it's implemented as a separate entry point in flyte propeller repo... If you're asking whether you can change it through config, the answer is no... It'll have to be a code change...

3/ would that be something that we could contribute? or we likely have to fork Flyte propeller

I think this is generic enough that others may use... I see no problem accepting that

ack. we will get back to you with a proposal doc if we are going with this route. Still exploring alternatives to implement our user-scoped workload isolation