Hey folks! Curious if your Flyte users have a specific folder structure to hold generalized tasks, workflows, etc? We're brainstorming what might be the best way to organize and isolate workflow dependencies. For example, some workflows could share the same Dockerfile & requirements, but others may have their own... etc... any best practices folks developed at their companies they're willing to share? We tried to follow something similar to Flytesnacks but I think it got a bit disorganized.

oh yeah please don’t follow flytesnacks. that repo is… overburdened. it needs to serve as 1) testing platform 2) code examples 3) outside-of-code documentation 4) probably something else i’m forgetting

Yes, actually my prompt is generally sparked by how complex our CI is getting.

Cc @freezing-boots-56761 / @elegant-australia-91422 and others please chime in

We're in the process of writing up our CI process for Flyte — actually think we've found a way to set it up that reduces a lot of boilerplate. The main aspect of our repo that allows us to do that is we're using a hermetic build system (Bazel, though one could probably achieve the same with pants or buck). We extended bazel in order to build a Flyte image and register entities

Super interesting @elegant-australia-91422! I'd be very interested into looking into this. We aren't using any tool at all right now, just some custom github actions we've written up for workflows that get pushed to our main branch. (and unfortunately right now sometimes that means a gha for each workflow as it gets updated 😞 )

ohh why a github action for each workflow?, why not package and register?

The github action is packaging up the workflow and registering them individually because we wanted the version number for each workflow to be tied to the git hash where it was updated.

But isn’t that the same?

Interestingly enough, using a build system like this makes it feasible to rebuild/push a fresh container image each time and re-register workflows