set the channel description: Channel for discussing Flyte & Bazel integration

Just set this up to discuss Bazel <> Flyte integration & any particulars that come up during development, as well as any work to get

rules_flyte

back up to date/upstream some of our own work

I'd be happy to contribute to

rules_flyte

. The implementation I put together this week is fairly encapsulated from our stack simply because its so new -- though that is a double edged sword

The only specifics we use are our internal py_binary/py_library targets. Historically we added linters as sidecar targets to that, though we’ve recently moved those to aspects. So we might be able to ditch the custom targets soon

Other than that we’re close to the same state

Joining late to the party, but we (Woven by Toyota/ex-level5) have also open sourced some parts of our monorepo.. Happy to conttibute/upstream our work as well.

Would love to contribute as well. We based much of our integration at stripe on the bazel rules but will need some time to remove the stripe specific components

Cc @Sam Eckert / @Rahul Mehta

One thing that'd be awesome to share is how each of our implementations is handling serialization, registration & execution. Do you all have separate targets for each, or are any of the steps handled together? In our case, we handle both serialization & registration in a single

.register

target, and handle execution w/ a

py_binary

target that takes in the launch plan name & user-specified CLI args that are passed to

flyte_remote.execute

inside that binary. One interesting concern is that it requires the user to remember to register prior to attempting to launch an execution, and this is an area we'd really like to clean up and make more seamless. Curious if anyone else has different solutions for this?

Folks remember Flyte slack is free. Good thing would be to put things in a doc

Hi Guys, I am using bazel to build the flyte target but when I use bazel output binaries to serialize and register the workflows it lead to ModuleNotFoundError: No module named 'pyflyte' at the time of task execution. The error occurred because the value of

resolver

in task container argument changed from usual

flytekit.core.python_auto_container.default_task_resolver

to

pyflyte.pypi_flytekit.site-packages.flytekit.core.python_auto_container.default_task_resolver

. Did anyone face this issue ? is there a way to control this?

@Sam Eckert also we figured out how to avoid the container_image step; since the entrypoint in our container delegates to the appropriate

pyflyte-*

command, just using the py3_image as-is worked (w/ the way that Flyte sets

args

for the pods)

This shaved ~45-60s off build time, which adds up over time

cc @James Sutton

Folks @Mohd Shahid Khan Afridi / @Rahul Mehta / @Sam Eckert can we get an example repo of Flyte with Bazel?

there are a lot of users who keep on asking

Yes, if there's any example Bazel build graphs that you guys are able to share, it would be greatly appreciated 🙇