@ancient-wolf-19325 With multiple propellers adjusted to one admin, each defined and managed separately on different k8 clusters can I make configuration so that propeller is based on selected project, not labels?

Yes, you can configure FlytePropeller sharding so that each FlytePropeller instance processes workflows for specific projects, rather than using only labels. The FlytePropeller Manager supports a "project" shard strategy, where you define a mapping of project IDs to specific FlytePropeller instances. Each instance will then process only the workflows for its assigned projects, even across different Kubernetes clusters, as long as they share the same FlyteAdmin control plane. This is achieved by setting the shard type to "Project" and specifying the per-shard project assignments in the manager configuration, using Kubernetes label selectors internally to enforce the mapping. See the configuration example in flytepropeller/manager/doc.go. Would you like more detail on how to set up this configuration? Sources: - flytepropeller/manager/doc.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 How do I apply this configuration - when I already have 2 clusters defined in configmap: clusters: clusterconfigs: in my helmchart? Ive defined clusters A and B that are remove as I had to define auth and k8s endpoint.

It seems likely that you can apply project-based sharding by configuring the FlytePropeller Manager with the "project" shard strategy, independent of your existing

clusters.clusterConfigs

(which define the remote clusters and their auth details). The sharding configuration is separate and should be set in the FlytePropeller Manager's config, mapping each project to a specific FlytePropeller instance, which in turn targets a specific cluster as defined in your

clusters.clusterConfigs

. The

labelClusterMap

in your clusters config is used for mapping project/domain/label to a cluster, while the sharding config in the manager determines which FlytePropeller instance processes which projects. See the example in flytepropeller/manager/doc.go. Would you like a step-by-step example of how to combine these configurations in your Helm chart? Sources: - flytepropeller/manager/doc.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 yes give me example

Certainly! Here’s a step-by-step example for combining project-based sharding with multiple clusters in your Helm chart: 1. Define your clusters in the

clusters.yaml

configmap (as you already have), specifying

clusterConfigs

for clusters A and B, including their auth and k8s endpoints. 2. Map projects to clusters using

labelClusterMap

in the same configmap, e.g., assign

project1

to cluster A and

project2

to cluster B. 3. Configure FlytePropeller Manager with the "project" shard strategy, assigning each project to a shard (i.e., a FlytePropeller instance). For example:

manager:
  pod-application: "flytepropeller"
  pod-template-name: "flytepropeller-template"
  pod-template-namespace: "flyte"
  scan-interval: 10s
  shard:
    type: Project
    per-shard-mapping:
      - ids:
          - project1   # This shard (FlytePropeller instance) handles project1
      - ids:
          - project2   # This shard (FlytePropeller instance) handles project2

4. Deploy FlytePropeller Manager with this configuration. Each FlytePropeller instance will process only the workflows for its assigned project(s), and the

labelClusterMap

will ensure those workflows are executed on the correct cluster. This setup ensures project-based sharding and cluster assignment are both respected. For more details, see the configuration examples in flytepropeller/manager/doc.go. Sources: - flytepropeller/manager/doc.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 but I want to select clusters based on project - is that possible?

Yes, you can select clusters based on project by using the

labelClusterMap

in your clusters config. You assign a label to each project, and then map that label to a specific cluster (or set of clusters with weights). When a workflow is launched under a project, FlyteAdmin uses the label to route executions to the correct cluster as defined in

labelClusterMap

. This allows you to direct all executions for a project to a specific cluster. See the example in clusters_config2_default_label.yaml. Would you like a concrete YAML example for this mapping? Sources: - clusters_config2_default_label.yaml 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 how do I assign so that workflows run in given project are given some labels by default? I dont want to specify labels, just projects in my workflows

You can assign default labels to all workflows in a given project by configuring project-level labels in FlyteAdmin. These labels are automatically applied to every workflow execution within that project, without needing to specify them in each workflow. This is managed in the control plane and does not require changes to your workflow code. See the implementation in execution_manager.go, where project labels are merged with workflow labels, with workflow labels taking precedence if there is a conflict. Would you like more detail on how to set or update these project-level labels? Sources: - execution_manager.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 yes, give me an example how to set read and update project-level labels

To set or update project-level labels in Flyte, use the

flytectl

CLI. You can create or update labels either inline or via a YAML file. For example, to update a project's labels inline:

flytectl update project --id my-project-id --labels team=data,env=prod

To update using a YAML file, create `project.yaml`:

id: "my-project-id"
labels:
  values:
    team: data
    env: prod

Then run:

flytectl update project --file project.yaml

To read (view) project labels, use:

flytectl get projects my-project-id -o yaml

This will display the current labels and other project metadata. Sources: - flytectl/cmd/update/project.go - flytectl/proposal/README.md - d400ebee - 231f9454 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 can you control those project based labels thru helmchart?

No, project-level labels in Flyte cannot be directly managed or updated through Helm charts. They must be set or updated using the

flytectl

CLI or FlyteAdmin API after deployment. Helm charts are used for deploying and configuring infrastructure, but project metadata like labels is managed separately at the control plane level. This limitation is discussed in flyte/issues/2069 and flyte/issues/2070. Would you like more detail on automating label management post-deployment? Sources: - flyte/issues/2069 - flyte/issues/2070 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 Ive modified the project with

flytectl update project --id flytesnacks --labels X=true

and the I have following labelClusterMap:

labelClusterMap:
        X:
          - id: X
            weight: 1
        Y:
          - id: Y
            weight: 1

I'd assume with that, every workflow submitted to

flytesnacks

goes to X, but from few submitted workflows one was routed to Y. Can you help me debug the problem?

Your expectation is correct: with the project label X=true and the labelClusterMap mapping X to cluster X, all workflows in flytesnacks should route to cluster X. If some are routed to Y, possible causes include: (1) workflow-level or domain-level execution cluster label overrides, (2) missing or misapplied project label, or (3) a fallback/default cluster label in your config. Flyte uses the most specific label available (workflow > domain > project), and if no label is found, it may randomly select a cluster from enabled ones, as described in flyte/issues/2882. To debug, check: - That the project label is correctly set (

flytectl get project flytesnacks -o yaml

). - That no workflow/domain-level execution cluster label overrides exist (

flytectl get execution-cluster-label -p flytesnacks -d <domain>

). - That your config does not specify a defaultExecutionLabel or fallback behavior. Would you like a step-by-step guide to check for overrides and config issues? Sources: - flyte/issues/2882 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 how do I remove execution-cluster-label?

To remove an execution-cluster-label, use the

flytectl delete execution-cluster-label

command. For example, to remove the label for a project and domain:

flytectl delete execution-cluster-label -p <project> -d <domain>

To remove the label for a specific workflow:

flytectl delete execution-cluster-label -p <project> -d <domain> <workflow_name>

You can also use the

--attrFile

option with a YAML file specifying the project, domain, and (optionally) workflow. See usage details in matchable_execution_cluster_label.go. Sources: - matchable_execution_cluster_label.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 how do I assign label to a specific workflow? I have defined workflow in python file and I run it using

pyflyte run --remote

To assign a label to a specific workflow execution when running with

pyflyte run --remote

, use the

--labels

option. For example:

pyflyte run --remote --labels key=value my_workflow.py my_workflow

This will apply the label to the workflow execution, and it will take precedence over project-level labels. For more details, see workflow_labels_and_annotations.md. Sources: - workflow_labels_and_annotations.md 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 sorry once again help me debug somethign. Ive labelClusterMap defined like that:

labelClusterMap:
        ride:
          - id: ride
            weight: 1
        solo:
          - id: solo
            weight: 1

and I launch workflows like that:

pyflyte run --remote --labels ride=true --project=flytesnacks sample_wf.py hello_world_wf

- yet, the workflows end up in both clusters. Am I doing it right? If so where there might be a problem with cluster selection

It seems likely that your usage is mostly correct, but there are a few areas to check. The label you pass with

--labels ride=true

will be attached to the execution, but Flyte's cluster selection logic matches the label key (not the key-value pair) to the

labelClusterMap

. So, the key should match exactly: if your

labelClusterMap

has

ride

, then the execution must have a label with key

ride

(the value is ignored for cluster selection). However, if you also have other labels (like

solo

), or if there are domain/project-level execution-cluster-label overrides, Flyte may route to other clusters. Additionally, if no matching label is found, Flyte may randomly select a cluster from the enabled set, as described in flyte/issues/2882. To debug: 1. Ensure no execution-cluster-label overrides exist for the project/domain/workflow (

flytectl get execution-cluster-label -p flytesnacks -d <domain>

). 2. Confirm that the label key in your command matches the

labelClusterMap

key exactly (e.g.,

--labels ride=anything

). 3. Check for a

defaultExecutionLabel

in your config, which could affect fallback behavior. Would you like a step-by-step guide to verify your label and config setup? Sources: - flyte/issues/2882 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 Ive confirmed no execution-cluster-label. Also confirmed that the labelClusterMap matches exactly..... also no defaultExecutionLabel. How to debug that further?

It seems likely that, if you have confirmed there are no execution-cluster-label overrides, your labelClusterMap matches exactly, and there is no defaultExecutionLabel, the next step is to verify what labels are actually attached to each workflow execution at runtime. Sometimes, labels may not be passed as expected, or there could be an issue in how pyflyte or FlyteAdmin interprets them. To debug further: 1. Check the execution metadata: After launching a workflow, use the Flyte UI or

flytectl get execution

to inspect the execution and confirm the labels present on the execution. This will show you exactly what labels FlyteAdmin received. 2. Review FlyteAdmin logs: Look for logs related to cluster selection and label matching. These logs may indicate why a particular cluster was chosen. 3. Check for label propagation issues: Ensure that the

--labels

flag is being parsed and sent correctly by pyflyte. Try running with a different label or with multiple labels to see if the behavior changes. 4. Validate cluster configs: Double-check that the cluster IDs in your

labelClusterMap

correspond to the actual cluster names in your

clusterConfigs

and that all clusters are enabled. If executions still route incorrectly, capturing the exact execution metadata and relevant FlyteAdmin logs will help pinpoint where the selection logic may be failing. Would you like more detail on how to inspect execution metadata or FlyteAdmin logs? Sources: - flyte/issues/2882 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 how do I remove label from project? Whats the flytectl update project command to clear out exisitn labels

It seems likely that to remove all labels from a project, you can use the

flytectl update project

command with an empty labels map. For example, using a YAML file:

id: "my-project-id"
labels:
  values: {}

Then run:

flytectl update project --file project.yaml

This will clear out all existing labels for the project. There is no direct CLI flag to remove all labels inline; the YAML method is the recommended approach. See usage in flytectl/cmd/update/project.go. Sources: - flytectl/cmd/update/project.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 I created that:

id: "flytesnacks"
labels:
  values: {}

then did this

flytectl update project --file x

and when I run

flytectl get projects flytesnacks -o yaml

I still see labels: values: things

If you still see

labels: values:

after updating with an empty map, this is expected: the

labels

field will remain in the project metadata, but the

values

map should be empty (i.e.,

labels: values: {}

), indicating no labels are set. The field itself is not removed, only its contents are cleared. This is the intended behavior as shown in flytectl/cmd/update/project.go. Would you like more detail on how Flyte represents empty labels in project metadata? Sources: - https://github.com/flyteorg/flyte/blob/master/flyteadmin/tests/project_test.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 No, I mean I still see some actual values. The thing I did before did not arase all the labels from project

If you still see actual label values after running

flytectl update project --file x

with an empty

values: {}

map, it is likely that the update did not overwrite the existing labels. This can happen if the update command or YAML file was not processed correctly, or if the CLI did not send an explicit empty map to FlyteAdmin. According to flyteadmin/tests/project_test.go, updating a project with an empty labels map should remove all labels, but if the map is omitted or not empty, old labels may persist. To ensure all labels are removed, double-check that your YAML file is formatted as:

id: "flytesnacks"
labels:
  values: {}

Then run:

flytectl update project --file x

If labels still persist, try explicitly setting the labels field to an empty map in the CLI or re-run the command, as sometimes the update may not propagate if the field is missing or not recognized as an overwrite. Would you like troubleshooting steps to verify the update or check for errors in the FlyteAdmin logs? Sources: - flyteadmin/tests/project_test.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 Ive confirmed everything, and still the labels are ignored when distributing workflows to specific clusters. Do I somehow need to enable that? I couldnt find anything In logs. where to look for help

If labels are being ignored for cluster selection despite correct configuration, ensure that the execution cluster label routing feature is enabled and supported in your Flyte deployment. This feature is standard in multi-cluster setups, but requires that FlyteAdmin, FlytePropeller, and your cluster config are all up-to-date and correctly referencing the

labelClusterMap

. There is no explicit "enable" flag, but misconfiguration or version mismatch can cause label-based routing to be ignored. Review your FlyteAdmin and FlytePropeller versions and confirm your config matches the documented structure in clusters_config2_default_label.yaml. For further help, check: - FlyteAdmin logs for messages about "execution cluster label" or "labelClusterMap" during execution creation. - That your FlytePropeller(s) are running and connected to the correct clusters. - That your Flyte version is at least 1.13.0, which includes recent improvements to executionClusterLabel support (CHANGELOG-v1.13.0.md). If the issue persists, consider opening a GitHub issue with your config and logs for community support. Sources: - clusters_config2_default_label.yaml - CHANGELOG-v1.13.0.md 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.