Hi all, I'm trying to get my implementation of a B...
# ask-the-community
s
Hi all, I'm trying to get my implementation of a Bazel/Flyte integration off the ground and I'm running into an issue on the last bit which is stumping me. I created a bazel macro called
flyte_library
(taking cues from @Rahul Mehta’s talk!). On run, that rule 1. Creates a py3_image with the workflow file, as well as pulling in the
aws
,
flytectl
, and
pyflyte-*
cli's. I wanted to keep things hermetic within the bazel env so I didn't create a base image with `awscli`/`pyflyte` pre-installed. 2. We add the
FLYTE_INTERNAL_IMAGE
tag, and then push this image to ECR. I'm still not 100% sure what
FLYTE_INTERNAL_IMAGE
does, but followed the examples I could find. 3. We then have a genrule which runs
docker run
using the image we just created, and calls a custom register script which wraps
pyflyte register
to register the workflow, and uses
flytectl
to enable/optionally execute any launchplans registered alongside the workflow. Registration works correctly as far as I can tell. The objects are created and viewable in the Console, but all tasks fail with:
Copy code
[1/1] currentAttempt done. Last Error: UNKNOWN::Outputs not generated by task execution
I can see the pod starting, pulling the correct image, and the
pyflyte-fast-execute
command exiting successfully via
kubectl
. No logs are created before the script exits so I'm having a bit of trouble identifying the issue. Weirder still, the exact same
pyflyte-fast-execute
command runs fine if I run it in a docker container locally.
I'm on
flytekit==1.2.7
btw
k
what i really wanted was to use python directly in bazel - but maybe thats not possible. otherwise -
flytekit.remote
might be cool
s
Yeah... I didn't see
FlyteRemote
until a few hours ago -- will probably switch over to that
r
Ah interesting, our
register
genrule just executes `pyflyte`/`flytectl` in a shell script. we also had to jump through a few hoops to actually construct a
pyflyte*-execute
entrypoint that worked w/ our py3_image, we wound up making an entrypoint that could launch pyflyte but include all of the workflow deps inside its runfiles tree
As for launching the workflow, we are using
FlyteRemote
to actually kick off the execution (we have an entrypoint which wraps both steps and shells out to bazel as a subprocess for the former) -- we don't hold the requirement that CLIs are fully hermetic
s
I basically wrap each of the
pyflyte*
scripts as a
py_binary
using the pip_parse/entry_point rules and then add symlinks to them in the container
r
Makes sense, that's a different approach to get to the same outcome
s
it may make sense that I remove that requirement -- but the fact it runs fine locally in the same image means the issue is probably something else
At least -- thats my assumption at the moment
Turns out this was because of the
ENTRYPOINT
being set in
py3_image
. Unsetting it solved the issue.
r
@Sam Eckert did you need to use the
py3_image
as the
base
in a normal
container_image
in order to unset the entrypoint?
s
Yeah thats right.
Kind of hacky but works fine
r
Yeah...in general it's an annoying amount of overhead, especially from a cold cache
Though we moved our base image build out of our bazel repo & just pull it in as a
WORKSPACE
dep, so that helped a bit
330 Views