Is there any declerative approach to flyte stuff l...
# ask-the-community
e
Is there any declerative approach to flyte stuff like gitopsy ?
k
Flyte is code declarative right? Question is what do you want to achieve
e
manage flyte via gitops
k
you mean things like quotas etc?
this can be using flytectl
but, if you are talking about flyte deployment itself - its helm and you will have to follow the regular guidance
s
I've also been thinking about this. Would a CRD to create and archive flyte projects be useful?
k
That could be
s
I’d be willing to give it a go if anyone thinks it’s worthwhile. I can start with a small POC.
k
But it might be an overkill
As the cluster resource controller does the job for now
Do you think it does not work for some cases?
s
At our org we try to manage most of our stuff through things like fluxcd/argocd so it would be nice to have it pulling from a CRD rather than us managing it through flytectl. Definitely just a nice to have at the moment, but would tie everything together
k
You don’t have to manage through flytectl right
You can apply a yaml to Flyte
r
I think this is referring to the entire "workflow" specification being able to be applied as a single CRD? This is an interesting tradeoff. We run data pipelines in Argo, & the
Workflow
/`CronWorkflow` manifests are easy to deploy/promote in a GitOps setting (ie. promotion involves copying the manifest from a staging namespace to a prod namespace, usually w/ some automation like a GitHub Action). We use Flux as well and it's a convenient pattern if you want to separate the repos where core workflow development vs deployment happen/enforce additional controls and disallow direct front-door access to the clusters the workload runs in
@Ketan (kumare3) is there a way to "apply" a manifest to handle 1) entity registration, and 2) executions?
s
That would be amazing as well.
k
Please draft an rfc, this would be amazing to learn
r
@Sam Cox this is becoming more important for us too. Let's connect on this?
k
And this is absolutely ok if you create one
r
Out of curiosity @Ketan (kumare3), how did Lyft handle this with Flyte (if you can share)? Seems like more stringent access controls/promotion story would've been important in an org that large?
k
Ya we had a separate deployment system - using Jenkins
s
My current scenario is we define our “projects” in one repo with namespace, service accounts, cross plane (ie s3), etc and the Flyte project has to be applied through flytectl. The main flux kustomization controller applied all these specs/CRDs.
k
When users would Promote workflows and that would allow registration sto new Domains
And service accounts were created in these ci pipelines
r
I see - so this was handled at the CD-level. We're currently following this approach as well (our actions runners can talk to the cluster that we run flyte in). But like @Sam Cox said, it'd be nice to have a consistent approach for both model training and data ETLs
s
@Rahul Mehta for sure.
r
Also @Sam Cox there seem to be a few different concepts here. Think this can be broken up into CRDs for 1) management and 2) registration of entities/execution
Both would add a lot of value for us, but can probably be tackled separately/are somewhat orthogonal
s
Yes 1. Is what I was going after but 2 would be a game changer
r
Nice - 1 is a no-brainer for us in terms of the value-add, and I've got some thoughts re: 2
s
Yes let’s connect. I’m just getting started in Flyte so I’m sure I haven’t seen all the pitfalls and use cases
r
Cool, let's schedule something next week. Can post an invite in #flyte-deployment in case other folks are interested in gitops-ifying their flyte deployments
s
Sounds good to me
By the way @Ketan (kumare3) love Flyte so far. It makes our life a lot easier
k
Ohh @Sam Cox pleasure to hear that. Thank you. One of my asks from the community is to share the word. Maybe write a blog or so
Also Sam please share your experience and what you want to improve at the community sync would be awesome/ rahul did it in December
s
That sounds good to me. I’d be happy to
k
Cc @Tyler Su / @David Espejo (he/him) can you please point Sam towards the signup sheet. Sam I think next dates are in April or may
Or @Samhita Alla
s
Hi folks. Was there any outcomes from the above meeting? We are interested in implementing a CRD/ operator for registering entities on a cluster
121 Views