Sam Eckert
01/27/2023, 1:19 AMflyte_library
(taking cues from @Rahul Mehta’s talk!). On run, that rule
1. Creates a py3_image with the workflow file, as well as pulling in the aws
, flytectl
, and pyflyte-*
cli's. I wanted to keep things hermetic within the bazel env so I didn't create a base image with `awscli`/`pyflyte` pre-installed.
2. We add the FLYTE_INTERNAL_IMAGE
tag, and then push this image to ECR. I'm still not 100% sure what FLYTE_INTERNAL_IMAGE
does, but followed the examples I could find.
3. We then have a genrule which runs docker run
using the image we just created, and calls a custom register script which wraps pyflyte register
to register the workflow, and uses flytectl
to enable/optionally execute any launchplans registered alongside the workflow.
Registration works correctly as far as I can tell. The objects are created and viewable in the Console, but all tasks fail with:
[1/1] currentAttempt done. Last Error: UNKNOWN::Outputs not generated by task execution
I can see the pod starting, pulling the correct image, and the pyflyte-fast-execute
command exiting successfully via kubectl
. No logs are created before the script exits so I'm having a bit of trouble identifying the issue. Weirder still, the exact same pyflyte-fast-execute
command runs fine if I run it in a docker container locally.flytekit==1.2.7
btwKetan (kumare3)
flytekit.remote
might be coolSam Eckert
01/27/2023, 1:31 AMFlyteRemote
until a few hours ago -- will probably switch over to thatRahul Mehta
01/27/2023, 1:34 AMregister
genrule just executes `pyflyte`/`flytectl` in a shell script. we also had to jump through a few hoops to actually construct a pyflyte*-execute
entrypoint that worked w/ our py3_image, we wound up making an entrypoint that could launch pyflyte but include all of the workflow deps inside its runfiles treeFlyteRemote
to actually kick off the execution (we have an entrypoint which wraps both steps and shells out to bazel as a subprocess for the former) -- we don't hold the requirement that CLIs are fully hermeticSam Eckert
01/27/2023, 1:39 AMpyflyte*
scripts as a py_binary
using the pip_parse/entry_point rules and then add symlinks to them in the containerRahul Mehta
01/27/2023, 1:40 AMSam Eckert
01/27/2023, 1:40 AMENTRYPOINT
being set in py3_image
. Unsetting it solved the issue.Rahul Mehta
01/29/2023, 5:48 PMpy3_image
as the base
in a normal container_image
in order to unset the entrypoint?Sam Eckert
01/29/2023, 5:54 PMRahul Mehta
01/29/2023, 5:56 PMWORKSPACE
dep, so that helped a bit