https://flyte.org logo
Title
r

Rahul Mehta

02/23/2023, 2:59 PM
One thing that'd be awesome to share is how each of our implementations is handling serialization, registration & execution. Do you all have separate targets for each, or are any of the steps handled together? In our case, we handle both serialization & registration in a single
.register
target, and handle execution w/ a
py_binary
target that takes in the launch plan name & user-specified CLI args that are passed to
flyte_remote.execute
inside that binary. One interesting concern is that it requires the user to remember to register prior to attempting to launch an execution, and this is an area we'd really like to clean up and make more seamless. Curious if anyone else has different solutions for this?
m

Mohd Shahid Khan Afridi

03/23/2023, 6:35 PM
Hi @Rahul Mehta, you got any lead here ? I am also looking forward to make the process one shot, without manually worrying about such dependency. Below is our way of doing it. we have also combined serialization & registration in a single
.register
target, My design for launchplan is to pass a list of launchplans defined in workflow file as an attribute to flyte_library rule and it will prepare a separate target for launchplan, which will have dependency on register. So flow will look like
.push
for image
.register
serialize and register (serialization dont produce any executable just the serialized proto consumed by registration)
.launch
activate the launcplans (deppends on registration but that will only build the executable for registeration will not run it) All the three steps will use the
commit-id
as version and dirty-repo will result in failure We have also ensured to make
pyflyte*
and
flytectl
as bazel dependency rather than installing separately in the dev environment