https://flyte.org logo
Join Slack
Powered by
<@U06RKAMK0UC> Please confirm that my understandin...
# ask-ai
m
@ancient-wolf-19325 Please confirm that my understanding of multicluster deployment cluster selection is correct: • In the config map I define the maping between execution-cluster-label and specific propeller. When the flag is provided, propeller is selected based on weights. • --ecl can be passed to workflow execution and it will be respected by the configmap • I can also pass the default ecl in the map, which will be used if no ECL flag is selected • Theres a way to use flytectl to set default ecl for each project and domain. Any of those points are not correct?
a
Your understanding is mostly correct: - You define the mapping between execution cluster labels (ECL) and clusters in the configmap (
labelClusterMap
), and weights determine selection if multiple clusters are mapped to the same label. The propeller is selected based on these weights if no explicit override is provided. - The 
--ecl
flag (execution cluster label) can be passed at workflow execution time (e.g., via 
pyflyte run --ecl
), and this label is respected, overriding project/domain defaults if present in the configmap. - You can set a 
defaultExecutionLabel
in the configmap, which is used if no ECL is specified at execution time and no project/domain override exists. - You can use 
flytectl update execution-cluster-label
to set a default ECL for each project and domain, which will be respected unless overridden at execution time. All your points are correct according to RFC: Select cluster execution when triggering an execution, Allow setting a ExecutionClusterLabel when triggering a Launchplan/Workflow/Task, flytectl get execution-cluster-label, and clusters_config2_default_label.yaml. Would you like more detail on any specific point? Sources: - RFC: Select cluster execution when triggering an execution - Allow setting a ExecutionClusterLabel when triggering a Launchplan/Workflow/Task - flytectl get execution-cluster-label - clusters_config2_default_label.yaml 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 thanks for confirmation. Now the question is: can I somehow force certain ecl flags for each project? so users ar enot allowed to overwrite them with --ecl? Also Can I somehow set the project-ecl connection on the helmchart level or I have to use dynamic API
a
It seems likely that, by default, users can override the project/domain ECL with the 
--ecl
flag at execution time; there is no explicit mechanism in the referenced sources to prevent this override. The system is designed so that per-execution ECL takes precedence over project/domain defaults set via configmap or 
flytectl
(RFC: Select cluster execution when triggering an execution). There is no documented way to "lock" a project/domain to a specific ECL and block user overrides. For setting project-ECL connections at the Helm chart level, there is community interest and some discussion about managing execution cluster labels via infrastructure-as-code (Helm values/configmap), but the standard method remains using 
flytectl update execution-cluster-label
or similar dynamic API calls (Execution Cluster label creation issue). Declarative management via Helm is not fully supported out-of-the-box, though recent PRs suggest progress in this area. Would you like more detail on possible workarounds or the current state of declarative ECL management? Sources: - RFC: Select cluster execution when triggering an execution - Execution Cluster label creation issue 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 yes please give me a ways for declarative ECL management and links to all discussions you can find about this problem on github
a
Declarative ECL (Execution Cluster Label) management is an active area of development in Flyte. As of now, the main approaches are: 1. Declarative Config in Helm/ConfigMap: Recent PRs propose supporting declarative management of matchable resources (including ECLs) via configuration files (e.g., Helm values or configmaps), not just imperative 
flytectl
commands. See Feat: Declarative management of matcheable resources and related RFC Declarative matchable resources. This allows you to specify ECL mappings in a YAML file and have them loaded at startup, but this is not yet the default or only method. 2. Imperative API (
flytectl
): The current standard is to use 
flytectl update execution-cluster-label --attrFile ...
to set ECLs for projects/domains. This is not fully declarative and can be overwritten by subsequent API calls or config changes (flytectl update matchable_execution_cluster_label.go). Key Discussions and Issues: - Feat: Declarative management of matcheable resources - RFC: Declarative matchable resources - Execution Cluster label creation issue - Allow flytectl to set a targetExecutionCluster - Add multi cluster set up to deployment section - Add support for multi cluster setup in helm Would you like a step-by-step example of how to use the declarative config approach? Sources: - Feat: Declarative management of matcheable resources - Execution Cluster label creation issue - flytectl update matchable_execution_cluster_label.go - Allow flytectl to set a targetExecutionCluster - Add multi cluster set up to deployment section - Add support for multi cluster setup in helm 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 so there is not a great way to define declaratively the execution cluster labels then I guess? How do I remove the ability for certain users to set configs by flytectl then at least
a
Correct, there is currently no fully mature, out-of-the-box way to manage execution cluster labels (ECLs) declaratively via Helm or configmap alone—though recent PRs are moving in that direction (Feat: Declarative management of matcheable resources, RFC: Declarative matchable resources). The standard method remains using 
flytectl
or the Admin API. To restrict users from setting or updating ECLs via 
flytectl
, you must control access to the FlyteAdmin API. This is typically done by limiting permissions at the network or authentication/authorization level, so only trusted users or service accounts can call the relevant Admin endpoints (Execution Cluster label creation issue). There is no built-in fine-grained RBAC for ECL updates in open-source Flyte; access control must be enforced externally. Would you like more detail on securing FlyteAdmin or on the current state of declarative ECL support? Sources: - Feat: Declarative management of matcheable resources - Execution Cluster label creation issue 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 any idea to build K8s job that gets run during deployment that sets the flyteclt those labels for me? or a cronjob that runs periodically and checks whether they match what I expect them to be? give me examples/ideas and yml files if you think its a good idea
a
Yes, you can use a Kubernetes Job or CronJob to run 
flytectl
and enforce execution cluster labels during deployment or periodically. This is a common workaround for environments lacking full declarative support. The Job would mount a config file (e.g., 
ecl.yaml
) and run 
flytectl update execution-cluster-label --attrFile ecl.yaml
. For periodic enforcement, use a CronJob. Example Kubernetes Job:
Copy code
apiVersion: batch/v1
kind: Job
metadata:
  name: set-flyte-ecl
spec:
  template:
    spec:
      containers:
      - name: flytectl
        image: cr.flyte.org/flyteorg/flytectl:v1.16.0
        command: ["flytectl", "update", "execution-cluster-label", "--attrFile", "/config/ecl.yaml"]
        volumeMounts:
        - name: config
          mountPath: /config
      restartPolicy: OnFailure
      volumes:
      - name: config
        configMap:
          name: flyte-ecl-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: flyte-ecl-config
data:
  ecl.yaml: |
    project: myproject
    domain: development
    value: myclusterlabel
Example Kubernetes CronJob:
Copy code
apiVersion: batch/v1
kind: CronJob
metadata:
  name: enforce-flyte-ecl
spec:
  schedule: "0 * * * *"  # every hour
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: flytectl
            image: cr.flyte.org/flyteorg/flytectl:v1.16.0
            command: ["flytectl", "update", "execution-cluster-label", "--attrFile", "/config/ecl.yaml"]
            volumeMounts:
            - name: config
              mountPath: /config
          restartPolicy: OnFailure
          volumes:
          - name: config
            configMap:
              name: flyte-ecl-config
This approach ensures your ECLs are set as desired and can revert unauthorized changes. For more, see flytectl update matchable_execution_cluster_label.go. Would you like more detail on error handling or idempotency for these jobs? Sources: - flytectl update matchable_execution_cluster_label.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 can you confirm the image you shared here flytectl is really existing? if not, what image should I use
a
It seems likely that the image 
cr.flyte.org/flyteorg/flytectl:v1.16.0
does not exist in the official Flyte container registries. There is no evidence in the data sources that a published 
flytectl
image is available; 
flytectl
is typically distributed as a binary, not a container image (flytectl README). You may need to build your own Docker image containing the 
flytectl
binary if you want to use it in a Kubernetes Job or CronJob. Would you like instructions or a sample Dockerfile for building a custom 
flytectl
image? Sources: - flytectl README 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 Is there any existing docker image that contains flytectl commandline which. Ican use to run the jobs you proposed above to control execution cluster labels?
a
The Flyte's documentation I have ingested does not mention any existing official Docker image that contains the 
flytectl
command-line tool ready for use in Kubernetes Jobs or CronJobs. Typically, 
flytectl
is distributed as a standalone binary for local installation, and there is no reference to a pre-built container image in the official documentation or deployment guides. If you need to use 
flytectl
in a containerized environment, you will likely need to build your own Docker image by starting from a base image (such as 
alpine
or 
ubuntu
), downloading the 
flytectl
binary from the official releases, and adding it to the image. If you have specific requirements for the image or need a sample Dockerfile, please provide more details about your environment or constraints. If this doesn't help, please reach out to the Flyte support team for further assistance. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 ive managed to get the job running, but my flytectl is set up with keycloak as auth. So my job fails due to the lack of auth. How do I fix that problem?
a
It seems likely that your Kubernetes Job fails because 
flytectl
requires authentication with Keycloak, and the Job lacks the necessary credentials or configuration to perform the OAuth2 flow non-interactively. Since 
flytectl
typically uses PKCE or device code flows that require browser interaction, running it in a non-interactive Job is challenging (flytectl README, Keycloak auth issues). A common workaround is to use a confidential client (service account) in Keycloak and configure 
flytectl
to use client credentials grant (if supported by your Flyte deployment). This involves: - Registering a confidential client in Keycloak for automation. - Providing the client ID and secret to the Job (as environment variables or mounted secrets). - Configuring 
flytectl
to use these credentials, typically by setting 
authType: ClientSecret
in the config file and referencing the secret. If your Flyte deployment does not support client credentials grant for 
flytectl
, you may need to pre-populate the token cache or use a service account with the required permissions (Keycloak config discussion). Would you like a step-by-step example for setting up 
flytectl
with client credentials in a Job? Sources: - Keycloak auth issues 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 can I somehow set the ececution labels form inside k8s without using flytectl? Like is there a configmap? Or anything I can use
a
No, you cannot set execution cluster labels (ECLs) directly from inside Kubernetes using a native ConfigMap or similar resource; Flyte does not natively support managing ECLs via a ConfigMap or Kubernetes resource. The standard and supported way is to use the FlyteAdmin API (e.g., via 
flytectl
) to set or update ECLs (Execution Cluster label creation issue). Declarative management via ConfigMap or Helm values is not yet fully supported, though there are ongoing discussions and PRs to add this feature (Feat: Declarative management of matcheable resources). Would you like more detail on the current state of declarative ECL management or possible workarounds? Sources: - Execution Cluster label creation issue 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 again, give me all possible workarounds. Auth within k8s job seems too complicated to set up
a
It seems likely that, since direct ConfigMap-based or declarative ECL management is not fully supported, and 
flytectl
authentication in a K8s Job is complex, your main workarounds are: 1. Pre-populate ECLs at Deployment Time: - Use 
flytectl
manually or in a CI/CD pipeline (outside the cluster) where authentication is easier to manage, to set the desired ECLs after deploying Flyte. - This can be scripted as part of your infrastructure-as-code or deployment process. 2. Custom Controller or Operator: - Build a lightweight custom Kubernetes controller or operator that watches a ConfigMap or CRD and calls the FlyteAdmin API (with a service account or pre-provisioned token) to enforce ECLs. - This requires some development effort but allows for declarative management. 3. Manual Enforcement: - Document the required ECLs and have an admin periodically check and enforce them using 
flytectl
from a secure, authenticated environment. 4. Use a Sidecar or Init Container with Pre-Authenticated Token: - If you can securely inject a pre-generated access token (with sufficient lifetime) into your Job (e.g., via a Kubernetes Secret), you can run 
flytectl
non-interactively. - This avoids interactive auth but requires secure token management and rotation. 5. Wait for Declarative Support: - Track and contribute to ongoing Flyte PRs/issues for declarative ECL management (Feat: Declarative management of matcheable resources), which may soon allow you to manage ECLs via Helm or ConfigMap. There is no built-in, fully declarative, or K8s-native way to manage ECLs as of now. All workarounds involve either external automation, custom code, or manual steps. Would you like more detail on any of these approaches? Sources: - https://github.com/flyteorg/flytesnacks/blob/master/examples/productionizing/productionizing/use_secrets.py - https://github.com/flyteorg/flyte/issues/6394 Note: I wasn’t able to find highly relevant data sources, but above are a few potentially relevant links. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
3 Views
Open in Slack
PreviousNext
Powered by