Rene Penkert01/17/2024, 11:08 AM
so that includes everything? and I can only scale everything together?
Worker is used a lot in regards to FlytePropeller but what is actually meant by that? An instantiation of one FlytePropeller aka Pod? A process as part of that
"FlytePropeller can scale to 1000s of workers on a single CPU"
Pod? A node as part of the cluster? What is a worker and how can I observe what it is doing?
3. How is scaling of the cluster supposed to work? Assume I want to increase the number of concurrent tasks. How would I make sure that the cluster can handle it? Scaling out FlyteAdmin & Scaling out Datacatalog & Scaling out FlytePropeller does not describe what I actually need to change to make it work except deploying the “FlytePropeller Manager”. I don’t have a deployment in my cluster that is called “FlyteAdmin” or “Datacatalog” so I’m not sure what is meant by “Datacatalog is a stateless service and its replicas (in the kubernetes deployment) can be simply increased to allow higher throughput”
Excuses in advance if these questions are trivial … . It would also help me if you can point me to some design documents or similar that I can read to answer my questions 🙂
Thomas Newton01/17/2024, 12:13 PM
and `flyte-core`https://docs.flyte.org/en/latest/deployment/deployment/index.html#helm. It sounds like you are using
. If you use
there will be separate deployments for flyteadmin, flytepropeller, datacatalog, etc.
I don't really know when its best to use
but if you want to scale to multiple kubernetes clsuters
is the only option.
Also "worker" means a goroutine within the flytepropeller process. There can be 1000s of workers per pod using just a few actual CPU cores.
Rene Penkert01/17/2024, 12:41 PM
David Espejo (he/him)01/17/2024, 12:57 PM
is effectively all the Flyte componentes packed, and flytepropeller can be sharded to accommodate more concurrent executions. Nevertheless,
while designed for envs with multiple K8s clusters, can be useful to connect that scale-out pattern to observable resources on K8s, like Pods
Rene Penkert01/17/2024, 1:01 PM
deployment is it sufficient to scale out the deployment of the binary or will that not have any effect?
David Espejo (he/him)01/17/2024, 2:05 PM
Rene Penkert01/17/2024, 2:08 PM
Dan Rammer (hamersaw)01/17/2024, 3:11 PM
Rene Penkert01/17/2024, 3:32 PM
to see the different components in action 😉
Might come back with some additional questions when I’m at that point 😜
Thanks a lot for the help 👍