Would it be possible to expose a `NodeSelector` fi...
# contribute
a
Would it be possible to expose a
NodeSelector
field in the
ExecutionSpec
in flyteidl/admin? Labels and annotations are already exposed
Our motivation from Stripe is pretty specific to our workload isolation goals, but I would imagine that there are many possible use cases for wanting to control a pod's scheduling
g
sure, would you mind creating a PR for this?
f
I don’t think we want to add this
1
You can use pod templates for this
Labels And annotations are very generic
a
hi @freezing-airport-6809 our use case is to pass dynamic values -- during execution time -- in this nodeSelector.. PodTemplate doesn't allow us to specify that. Podtemplate is static.
Adding on to what Vanshika shared.. User wants to specify how the pod is scheduled on which node.. When they use Flytekit to launch programmatically, i think people naturally want to specify node-selector for their workflows.
Do you suggest other alternative mechanism that allows this basic scheduling requirements?
f
How does the user know about the nodes? This will lead to non reproducible and weird behavior. My suggestion is to use pod template names in the back
a
Our use case • In organization that requires some workload segmentation, there is necessary contract between user and compute infra. Eg. Pods of
teamA
will have nodeSelector
<http://company.com/team|company.com/team>: teamA
and the node should have a corresponding label. ◦ This is actually Karpenter standard pattern "to provide workload segregation across nodes", where provisioner will create new nodes with label corresponding to pending pods value of the
NodeSelector
• We apply nodeSelector at execution time, and expect the scheduling behavior to be per execution. Essentially, we are ok with different registered workflow/task having different nodeSelector. (Unlike pod template name) • We have admin-proxy backend, where we have a reliable user identity and we would like to inject NodeSelector into the Execution request spec (similarly to how labels/annotation customizable on Flyte UI is applied to all tasks) Using pod template names • i know Dan works with the community for this useful feature, but i think it'll be hard for us.. ◦ we want nodeSelector per user, so we will need to create that many number of podTemplates when new users in the company get onboarded.. ◦ Pod template name is per-task, and we want to apply this scheduling for all tasks in a workflow. ◦ More importantly, users can choose any podTemplate meant for other users, so for our use case to achieve security objectives on user-specific workload isolation, during runtime, i can specify pod template for nodeSelector of any other users
c
I don't think adding node selectors at that layer is what you want. Not all plugins allow flyte to customize the pods, fully, that end up getting created.. I think what you really want is a single pod template (default) that says "nodeselector: pod.labels.user_identity" Assuming flyte passes down the user principal as a trusted label.. 1. I think it's reasonable to add that label in flyteadmin (if you haven't already done that in your proxy) 2. K8s doesn't allow downstream APIs (referencing pod labels) in node selectors I don't think.. but this can be achieved through an admission webhook (flyte's or kyverno or others)
a
Not all plugins allow flyte to customize the pods, fully, that end up getting created..
1/ i see. Would "labels" added as part of execution request also get propagated to all plugins? But what you are saying is rather, the nodeSelector is less general than labels such that the selector might be dropped in some cases?
pod template (default) that says "nodeselector: pod.labels.user_identity"
...
K8s doesn't allow downstream APIs (referencing pod labels) in node selectors
2/ Just to confirm, you are saying
nodeselector: pod.labels.user_identity
as in idea right? Implementation-wise, you are suggesting the admission webhook.
admission webhook (flyte's )
3/ Are there ways we can add on to Flyte's admission webhook?
c
1/ exactly... 2/ you got it! 3/ it's implemented as a separate entry point in flyte propeller repo... If you're asking whether you can change it through config, the answer is no... It'll have to be a code change...
a
3/ would that be something that we could contribute? or we likely have to fork Flyte propeller
c
I think this is generic enough that others may use... I see no problem accepting that
a
ack. we will get back to you with a proposal doc if we are going with this route. Still exploring alternatives to implement our user-scoped workload isolation
thanks Haytham and Ketan!