Hello, I also have a question about scaling but on...
# ask-the-communityd
David Cupp
12/21/2022, 4:13 PMHello, I also have a question about scaling but on a different axis -- what is the expected way to scale Flyte servers horizontally. I can see that scaling the workers can be done by having multiple K8S clusters, but if I have millions or billions of workflows and the primary database and server get overwhelmed, is there a way to shard them? Or do I simply need to have multiple Flyte deployments at that point?
k
Ketan (kumare3)
12/21/2022, 4:56 PM
At the moment the core reader database is single
Ketan (kumare3)
12/21/2022, 4:56 PM
But remember the entire state is stored in etcd, so as you add more k8s clusters you are indeed sharding the state db
Ketan (kumare3)
12/21/2022, 4:57 PM
We think large number of workflows at runtime only impact the state store, if you need you can share the execution storage service, but that would be needed at very high rate
Ketan (kumare3)
12/21/2022, 4:58 PM
Happy to jump on a call to discuss
d
David Cupp
12/21/2022, 5:07 PMAwesome thanks. I think this is all I need.
163 Views