https://flyte.org logo
#announcements
Title
# announcements
r

Rahul Mehta

08/09/2022, 9:16 PM
Hey all, we're moving forward with our migration to flyte and are making sure that various end-user workflows are at parity w/ our current Kubeflow deployment. A couple of topics that have come up include the best ways to support multi-tenancy/projects for experimentation. 1. We currently group runs submitted by various users as "experiments" in Kubeflow. I notice in Flyte there is an option in the UI to filter down to only workflows submitted by the currently-authenticated user, but more broadly what's the best way to group different sets of experiments that multiple users are running 2. Does having a project per user make sense (ie. for a user's dev workspace)? I noticed in the docs that a common pattern is having one project per repo, but in our current setup we run everything out of one monorepo.
❤️ 3
k

Ketan (kumare3)

08/10/2022, 3:54 AM
So projects are very light weight, though I do not think project per user make sense. At a point at lyft we had 400 projects, and I think Spotify has around that number
Projects absolutely make sense to group experiments by multiple users, but more correctly put, it is a logical grouping to indicate one line of business or one line of work that can span one or more teams - also repos. Most folks as you said it as a proxy of a repo
Even within a mononrepo you can group sub folders into a project. Cc @varsha Parthasarathy they use a mono repo too
v

varsha Parthasarathy

08/10/2022, 4:00 AM
@Rahul Mehta = Yes, we infamously use
monorepo
and each sub-folder (each of our autonomy teams) are grouped into a flyte project with their iam/s3/ml permissions.. . Let me know how I can help 🙏
r

Rahul Mehta

08/10/2022, 1:13 PM
@varsha Parthasarathy that makes sense for sub-folders in the monorepo to correspond to projects; with a little refactoring I think we could achieve this. For projects where multiple users are running experiments w/ many executions, how do the navigate the UI/find executions, etc? There didn't immediately seem to be a way to tag executions so that they'd be discoverable in the UI
k

Ketan (kumare3)

08/10/2022, 1:30 PM
You can name them, but yes there is no tagging today. I think that is a fair feature request
v

varsha Parthasarathy

08/10/2022, 3:48 PM
@Rahul Mehta - So in our mono-repo : has bazel integration .. So when users want to register/build a wf, they run a
bazel
command - while doing that, we generate an
exec id
and share the link to users. Something like this:
r

Rahul Mehta

08/10/2022, 4:29 PM
Got it, we're using
bazel
as well so that approach makes a lot of sense. Do you have users commit the workflows before running typically, & if so do you segment the monorepo between research workflows vs core development?
v

varsha Parthasarathy

08/10/2022, 4:52 PM
have users commit the workflows before running typically,
Yes, we made it a pre-req in bazel builds.. If someone wants to “push” their wf code to artifactory, they have to commit it.
👍 1
the monorepo between research workflows vs core development
Absolutely.. We achieved this using K8s namespace separation:
Dev vs prod.
. A lot of poc/research stuff would be in dev and when someone wants to make this image a stable one, we have setup a few pipelines where they can move them to prod.
Awesome that you use bazel 🙂
i’m oncall this week, so expect delay in responses, keep your questions coming, will share answers .. 👌
🙏 1
82 Views