This message was deleted.

Yes, you can configure FlytePropeller sharding so that each propeller instance manages workflows for specific projects, rather than using labels. This is achieved by setting the shard strategy to "Project" in the FlytePropeller manager configuration, and specifying a per-shard mapping of project IDs to each propeller instance. Each propeller will then use Kubernetes label selectors to process only the FlyteWorkflows for its assigned projects, ensuring project-based sharding without relying on custom labels. See the configuration example in the official documentation for details on the "project" shard type and per-shard mapping usage. Would you like more detail on how to set up the configuration or example YAML? Sources: - flytepropeller/manager/doc.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 where in flyte-core helmchart do I define it?

In the flyte-core Helm chart, you define project-based sharding for FlytePropeller under the

core.manager

section in your values YAML file (e.g.,

values-eks.yaml

). Set

flytepropeller.manager: true

and configure

core.manager.shard

with

type: Project

and the

per-shard-mapping

list to assign projects to shards. This configuration is merged into the deployment via Helm. Would you like a concrete YAML example for this setup? Sources: - flytepropeller/manager/doc.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 As far as I understand, the documentation you shared is about sharding singluar flyte workflows. I however want a selection of propeller being done by selecting project, not shard singluar workflow

You can achieve project-based selection of FlytePropeller instances by using the "Project" shard strategy in the FlytePropeller manager configuration. This approach assigns entire projects (not individual workflows) to specific propeller instances, so each instance processes all workflows for its assigned projects. The manager uses Kubernetes label selectors to ensure each propeller only handles FlyteWorkflows with the specified project label, effectively routing all workflows from a project to a designated propeller instance. This is not per-workflow sharding, but project-level assignment. Would you like a step-by-step example of how to configure this in your Helm values file? Sources: - flytepropeller/manager/doc.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 yes give me step by step example, also point out which configuration is done on admin cluster vs each of the propellers

It seems likely that you can set up project-based FlytePropeller sharding by configuring the FlytePropeller manager in your Helm values file, with most configuration done on the admin/control plane cluster. Here’s a step-by-step example: 1. On the Admin (Control Plane) Cluster: - In your

values.yaml

(or a custom values file), set

flytepropeller.manager: true

to enable the manager. - Under

core.manager

, define the sharding strategy as follows:

flytepropeller:
     manager: true

   core:
     manager:
       pod-application: "flytepropeller"
       pod-template-container-name: "flytepropeller"
       pod-template-name: "flytepropeller-template"
       pod-template-namespace: "flyte"
       scan-interval: 10s
       shard:
         type: Project
         per-shard-mapping:
           - ids:
               - project1
           - ids:
               - project2
           - ids:
               - "*"

- This configuration ensures that one propeller instance handles all workflows for

project1

, another for

project2

, and a third for all other projects. 2. On Each Propeller (Data Plane) Cluster: - No special sharding configuration is needed on the individual propeller clusters; the manager in the admin cluster will launch and manage the propeller pods with the correct label selectors based on the above configuration. 3. Deploy the Helm Chart: - Apply your Helm chart with the updated values file to the admin cluster. The manager will create and manage the propeller pods as specified. This setup ensures project-based routing of workflows to propeller instances, with all sharding logic centralized in the admin cluster’s configuration. Sources: - flytepropeller/manager/doc.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 can you explain the per-shard-mapping? what is "ids" and how does it select which propeller is being used by which project?

The

per-shard-mapping

is a list that assigns specific project IDs to each FlytePropeller instance (shard) when using the "Project" shard strategy. The

ids

field under each mapping specifies which projects a particular propeller instance will manage. Each propeller is configured with Kubernetes label selectors to only process FlyteWorkflows with the matching project label, so all workflows for a given project are routed to the designated propeller instance. For example, if you set:

per-shard-mapping:
  - ids: [project1]
  - ids: [project2, project3]
  - ids: ["*"]

the first propeller handles

project1

, the second handles

project2

and

project3

, and the third handles all other projects not explicitly listed. This ensures project-level assignment of workflows to propeller instances. Would you like more detail on how the label selectors work or how to verify the assignment? Sources: - flytepropeller/manager/doc.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 how do I know which propeller is first, secound etc? where is the order being defined?

The order of propeller instances is determined by the order of entries in the

per-shard-mapping

list. The first entry corresponds to the first propeller, the second entry to the second propeller, and so on. Each propeller is launched by the manager with a deterministic name and assigned the projects listed in its corresponding

ids

entry. The mapping is positional: the first mapping in the list is for the first propeller, the second for the second, etc. Would you like to see how the pod names or selectors are generated for each propeller? Sources: - flytepropeller/manager/doc.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 but how do I know which propeller is defined as first one? I define propellers in configmap: clusters: clusterConfigs. Does it correspond to the order of that entry in values file? If I have propeller X and Y how do I define which is 1st and which is 2nd

My understanding is that when using the FlytePropeller manager with project-based sharding, the order of the

per-shard-mapping

list in the manager configuration determines the assignment, not the order of entries in

clusters: clusterConfigs

or the names X and Y you might use elsewhere. The manager itself creates and manages the propeller pods, assigning each one to the projects listed in the corresponding position in

per-shard-mapping

. The first entry in

per-shard-mapping

is assigned to the first managed propeller pod, the second to the second, and so on—this is independent of any external cluster naming. If you are manually defining propeller deployments (not using the manager), project-based sharding as described is not supported; the manager must control the assignment. The mapping is positional and handled internally by the manager, not by the order in

clusterConfigs

. Would you like more detail on how the manager names or launches the pods? Sources: - flytepropeller/manager/doc.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.