https://flyte.org logo
#flyte-deployment
Title
# flyte-deployment
e

Ethan Brown

11/15/2023, 7:05 PM
Hi folks - doing a little research right now on various pipeline implementations (Kubeflow / Argo / Prefect amongst others). I'm digging into the details around multi-cluster setup / configuration and overall architecture. Hopefully I'm in the right place! I was a little surprised to see so many k8s primitives exposed in the install guide -- https://docs.flyte.org/en/latest/deployment/deployment/multicluster.html#control-plane-configuration Has there been any other feedback around that or intention to make the agent config any simpler? Are there similar guides for AKS, GKE and on-prem? (I should be able to sort out the infra differences, but prior art would save me time) TIA!
d

David Espejo (he/him)

11/15/2023, 7:27 PM
Hi @Ethan Brown and welcome to the Flyte community Multi-cluster is certainly the most advanced deployment option and it's true that currently it requires to deal with multiple K8s resources. There's already an Issue open in regards to enabling the
flyte-core
chart (the one used for multicluster) to consume pre-existing secrets, facilitating a bit more of "separation of concerns" in that step. This repo is where we're starting to add reference implementations with TF modules (currently for EKS, GKE soon-to-be-merged). Also there's a semi-manual deployment tutorial available here For AKS we don't have a guide just yet, the team at #azure-wg has been active improving the compatibility and sharing knowledge, but we probably need now to collect this knowledge on a single place. I hope any of this is useful for you
e

Ethan Brown

11/15/2023, 7:42 PM
Thanks a bunch @David Espejo (he/him) - that's very helpful.
Let me dig into that a bit more -- is there a good place to further discuss / post ideas or concerns about the infra architecture? I think in my ideal world, the propeller would operate similarly to a self-hosted CI agent like you see in Circle CI, Azure DevOps or GitHub Actions -- that is, the agent uses a PAT to register itself as part of a work pool the server defines. The server never makes inbound connections to agents (it looks like comms are bi-directional in Flyte?) -- agents long poll the server to determine if they need to schedule work. Scheduling happens periodically as agents pick up jobs from the server based on worker metadata matching requests.
d

David Espejo (he/him)

11/15/2023, 8:14 PM
As for long-living posts, Github Discussions are probably better than Slack. For sync discussions we also have the biweekly Contributors meetup where you can chat to maintainers. In regards to Flyte architecture, FlyteAdmin submits the Workflow CR create/update operation to the K8s API and the flytepropeller controller watches and reconciles resources in K8s, reporting execution status back to flyteadmin. So from control plane->data plane is more an indirect path.
e

Ethan Brown

11/15/2023, 8:35 PM
Great - thanks for that! I understand the model better now
5 Views