This message was deleted.
# ask-ai
e
This message was deleted.
a
Yes, you can configure FlytePropeller sharding so that each propeller instance manages workflows for specific projects, rather than using labels. This is achieved by setting the shard strategy to "Project" in the FlytePropeller manager configuration, and specifying a per-shard mapping of project IDs to each propeller instance. Each propeller will then use Kubernetes label selectors to process only the FlyteWorkflows for its assigned projects, ensuring project-based sharding without relying on custom labels. See the configuration example in the official documentation for details on the "project" shard type and per-shard mapping usage. Would you like more detail on how to set up the configuration or example YAML? Sources: - flytepropeller/manager/doc.go πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 where in flyte-core helmchart do I define it?
a
In the flyte-core Helm chart, you define project-based sharding for FlytePropeller under the
core.manager
section in your values YAML file (e.g.,
values-eks.yaml
). Set
flytepropeller.manager: true
and configure
core.manager.shard
with
type: Project
and the
per-shard-mapping
list to assign projects to shards. This configuration is merged into the deployment via Helm. Would you like a concrete YAML example for this setup? Sources: - flytepropeller/manager/doc.go πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 As far as I understand, the documentation you shared is about sharding singluar flyte workflows. I however want a selection of propeller being done by selecting project, not shard singluar workflow
a
You can achieve project-based selection of FlytePropeller instances by using the "Project" shard strategy in the FlytePropeller manager configuration. This approach assigns entire projects (not individual workflows) to specific propeller instances, so each instance processes all workflows for its assigned projects. The manager uses Kubernetes label selectors to ensure each propeller only handles FlyteWorkflows with the specified project label, effectively routing all workflows from a project to a designated propeller instance. This is not per-workflow sharding, but project-level assignment. Would you like a step-by-step example of how to configure this in your Helm values file? Sources: - flytepropeller/manager/doc.go πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 yes give me step by step example, also point out which configuration is done on admin cluster vs each of the propellers
a
It seems likely that you can set up project-based FlytePropeller sharding by configuring the FlytePropeller manager in your Helm values file, with most configuration done on the admin/control plane cluster. Here’s a step-by-step example: 1. On the Admin (Control Plane) Cluster: - In your
values.yaml
(or a custom values file), set
flytepropeller.manager: true
to enable the manager. - Under
core.manager
, define the sharding strategy as follows:
Copy code
flytepropeller:
     manager: true

   core:
     manager:
       pod-application: "flytepropeller"
       pod-template-container-name: "flytepropeller"
       pod-template-name: "flytepropeller-template"
       pod-template-namespace: "flyte"
       scan-interval: 10s
       shard:
         type: Project
         per-shard-mapping:
           - ids:
               - project1
           - ids:
               - project2
           - ids:
               - "*"
- This configuration ensures that one propeller instance handles all workflows for
project1
, another for
project2
, and a third for all other projects. 2. On Each Propeller (Data Plane) Cluster: - No special sharding configuration is needed on the individual propeller clusters; the manager in the admin cluster will launch and manage the propeller pods with the correct label selectors based on the above configuration. 3. Deploy the Helm Chart: - Apply your Helm chart with the updated values file to the admin cluster. The manager will create and manage the propeller pods as specified. This setup ensures project-based routing of workflows to propeller instances, with all sharding logic centralized in the admin cluster’s configuration. Sources: - flytepropeller/manager/doc.go πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 can you explain the per-shard-mapping? what is "ids" and how does it select which propeller is being used by which project?
a
The
per-shard-mapping
is a list that assigns specific project IDs to each FlytePropeller instance (shard) when using the "Project" shard strategy. The
ids
field under each mapping specifies which projects a particular propeller instance will manage. Each propeller is configured with Kubernetes label selectors to only process FlyteWorkflows with the matching project label, so all workflows for a given project are routed to the designated propeller instance. For example, if you set:
Copy code
per-shard-mapping:
  - ids: [project1]
  - ids: [project2, project3]
  - ids: ["*"]
the first propeller handles
project1
, the second handles
project2
and
project3
, and the third handles all other projects not explicitly listed. This ensures project-level assignment of workflows to propeller instances. Would you like more detail on how the label selectors work or how to verify the assignment? Sources: - flytepropeller/manager/doc.go πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 how do I know which propeller is first, secound etc? where is the order being defined?
a
The order of propeller instances is determined by the order of entries in the
per-shard-mapping
list. The first entry corresponds to the first propeller, the second entry to the second propeller, and so on. Each propeller is launched by the manager with a deterministic name and assigned the projects listed in its corresponding
ids
entry. The mapping is positional: the first mapping in the list is for the first propeller, the second for the second, etc. Would you like to see how the pod names or selectors are generated for each propeller? Sources: - flytepropeller/manager/doc.go πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 but how do I know which propeller is defined as first one? I define propellers in configmap: clusters: clusterConfigs. Does it correspond to the order of that entry in values file? If I have propeller X and Y how do I define which is 1st and which is 2nd
a
My understanding is that when using the FlytePropeller manager with project-based sharding, the order of the
per-shard-mapping
list in the manager configuration determines the assignment, not the order of entries in
clusters: clusterConfigs
or the names X and Y you might use elsewhere. The manager itself creates and manages the propeller pods, assigning each one to the projects listed in the corresponding position in
per-shard-mapping
. The first entry in
per-shard-mapping
is assigned to the first managed propeller pod, the second to the second, and so onβ€”this is independent of any external cluster naming. If you are manually defining propeller deployments (not using the manager), project-based sharding as described is not supported; the manager must control the assignment. The mapping is positional and handled internally by the manager, not by the order in
clusterConfigs
. Would you like more detail on how the manager names or launches the pods? Sources: - flytepropeller/manager/doc.go πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.