https://flyte.org logo
Join Slack
Powered by
Hello all, We are in the midst of evaluating open...
# flyte-support
c
Hello all, We are in the midst of evaluating open source ML pipeline and workflow orchestration tools w/in our organization. I received lots of great recommendations pointing me towards Flyte and am interested in doing a POC at some point. I'm still learning more and more about Flyte by watching some webinars. I did have some questions about multi-tenancy because that seems to be the biggest pain point when adopting new tools
We have multiple Data Science teams, each with their own respective AWS accounts. Does Flyte encourage deploying multiple instances of the infrastructure in our stakeholders' accounts or does it work best by having a centralized instance in own designated Machine Learning Platform AWS account?
t
hey @creamy-honey-22699 welcome!
you can certainly deploy one per aws account if that’s what they already have. most companies only have one aws account where their ml platform and their ml workflows live.
are these separate top level accounts or subaccounts?
i don’t think i’ve actually come across this style of use-case before. the main thing to keep in mind is data and data transfer.
i think the answer to this question will mostly depend on that. one of flyte’s advantages is that it has multi-tenancy features built in, so you don’t have to manage things yourself. but that said if you’re going to incur the price cost and time cost of a lot of data transfer as ml workloads pull and push large amounts of data, then that’s certainly something to consider as well.
c
Thank you @thankful-minister-83577 for your helpful information! Yes it's a very unique paradigm. To answer your question, yes these are separate top level accounts, each with their own account signature (we call this an account moniker) and prod, staging and dev environments. However, data would typically live in a single account that we call the Data Lab. You do bring up a good point that each of these different accounts have their own billing so model training and compute should be within their respective accounts because we don't want to incur the cost for that. I'm wondering if it's possible to have the UI and server in our centralized account but somehow do the compute on the customer accounts using Flyte. Is there any documentation you can point me to that delves into multi-tenancy details more in depth and also deployment patterns?
t
it is possible to have a multi-cluster setup yes. https://docs.flyte.org/en/latest/deployment/multicluster.html (btw, there’s an active pr out that moves some of these articles around, so the location will change in the next couple days)
that article there guides the user through just that, how to run multiple control planes.
c
I'll take a look, thank you. Appreciate the guidance. And multi-cluster allows you to spin up cross-account clusters?
t
this paradigm is meant for very large scale deploys and is what we used at lyft. but this was only ever done in one account, not two
so yeah not sure about that last question
and keep in mind there is still data that flows from the data plane to the control plane and vice versa.
c
It's tricky because these are essentially separate accounts with different IAM roles and such.
t
rpc calls to the operator, event information, and what we call metadata input/output (primitives like strings and floats, so flyte can display them on the ui)
👍 1
offloaded data (like dataframes and files) will remain in the target location but the address of that location will be sent back for instance.
different deployments of propeller can have a different default location for this offloaded data. and this setting is also configurable at the project/domain/workflow level
it is indeed tricky… can’t promise it’ll work, you’ll almost certainly have to adapt the helm chart
c
Got it thanks - yeah that might be a lot of toil. It might be better to just go with the deploy 1 to N account(s) approach then. I'll keep this in mind though.
t
let us know how it goes!
happy to help debug anything that might arise from this approach
🙏 1
156 Views
Open in Slack
PreviousNext
Powered by