crooked-artist-67935
01/27/2023, 1:19 AMflyte_library
(taking cues from @elegant-australia-91422ās talk!). On run, that rule
1. Creates a py3_image with the workflow file, as well as pulling in the aws
, flytectl
, and pyflyte-*
cli's. I wanted to keep things hermetic within the bazel env so I didn't create a base image with `awscli`/`pyflyte` pre-installed.
2. We add the FLYTE_INTERNAL_IMAGE
tag, and then push this image to ECR. I'm still not 100% sure what FLYTE_INTERNAL_IMAGE
does, but followed the examples I could find.
3. We then have a genrule which runs docker run
using the image we just created, and calls a custom register script which wraps pyflyte register
to register the workflow, and uses flytectl
to enable/optionally execute any launchplans registered alongside the workflow.
Registration works correctly as far as I can tell. The objects are created and viewable in the Console, but all tasks fail with:
[1/1] currentAttempt done. Last Error: UNKNOWN::Outputs not generated by task execution
I can see the pod starting, pulling the correct image, and the pyflyte-fast-execute
command exiting successfully via kubectl
. No logs are created before the script exits so I'm having a bit of trouble identifying the issue. Weirder still, the exact same pyflyte-fast-execute
command runs fine if I run it in a docker container locally.crooked-artist-67935
01/27/2023, 1:22 AMflytekit==1.2.7
btwfreezing-airport-6809
flytekit.remote
might be coolcrooked-artist-67935
01/27/2023, 1:31 AMFlyteRemote
until a few hours ago -- will probably switch over to thatelegant-australia-91422
01/27/2023, 1:34 AMregister
genrule just executes `pyflyte`/`flytectl` in a shell script. we also had to jump through a few hoops to actually construct a pyflyte*-execute
entrypoint that worked w/ our py3_image, we wound up making an entrypoint that could launch pyflyte but include all of the workflow deps inside its runfiles treeelegant-australia-91422
01/27/2023, 1:39 AMFlyteRemote
to actually kick off the execution (we have an entrypoint which wraps both steps and shells out to bazel as a subprocess for the former) -- we don't hold the requirement that CLIs are fully hermeticcrooked-artist-67935
01/27/2023, 1:39 AMpyflyte*
scripts as a py_binary
using the pip_parse/entry_point rules and then add symlinks to them in the containerelegant-australia-91422
01/27/2023, 1:40 AMcrooked-artist-67935
01/27/2023, 1:40 AMcrooked-artist-67935
01/27/2023, 1:40 AMcrooked-artist-67935
01/27/2023, 10:59 AMENTRYPOINT
being set in py3_image
. Unsetting it solved the issue.elegant-australia-91422
01/29/2023, 5:48 PMpy3_image
as the base
in a normal container_image
in order to unset the entrypoint?crooked-artist-67935
01/29/2023, 5:54 PMcrooked-artist-67935
01/29/2023, 5:55 PMelegant-australia-91422
01/29/2023, 5:56 PMelegant-australia-91422
01/29/2023, 5:56 PMWORKSPACE
dep, so that helped a bit