Hi everyone! :wave: I’m attempting to register an ...
# ask-the-communityb
Bradley Worley
03/09/2023, 6:44 PMHi everyone! 👋 I’m attempting to register an imperative workflow for remote execution, using a bazel build environment, and I’m seeing a failure I don’t completely understand
(MWE and error in the thread)
Bradley Worley
03/09/2023, 6:45 PMThe process I’m going through:
Copy code
from flytekit import Workflow
from flytekit.configuration import (
Config,
SerializationSettings,
)
from flytekit.remote import FlyteRemote
remote = FlyteRemote(
config=Config.auto(config_file=MY_CONFIG_PATH),
default_project=MY_PROJECT,
default_domain=MY_DOMAIN,
data_upload_location=MY_S3_BUCKET,
)
workflow = Workflow(f"my.workflows.{wf_name}")
# build out the workflow ...
ss = SerializationSettings.for_image(
image=f"{DOCKER_HOST}/flyte_workflow:latest",
version=version_str,
project=MY_PROJECT,
domain=MY_DOMAIN,
)
remote.register_workflow(
workflow,
serialization_settings=ss,
)
execution = remote.execute(
workflow,
inputs=inputs,
wait=False,
project=MY_PROJECT,
domain=MY_DOMAIN,
)Bradley Worley
03/09/2023, 6:45 PMAnd the error:
Copy code
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f52a73cdf369d4ded82b-n0-0] terminated with exit code (1). Reason [Error]. Message:
ask_cmd
_execute_task(
File "/app/pip_pypi__flytekit/flytekit/exceptions/scopes.py", line 160, in system_entry_point
return wrapped(*args, **kwargs)
...
File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'melody'Bradley Worley
03/09/2023, 6:48 PMIt seems like the issue is that
pyflyte-execute--resolver Copy code
melody.pip_pypi__flytekit.flytekit.core.python_auto_container.default_task_resolver Copy code
app.pip_pypi__flytekit.flytekit.core.python_auto_container.default_task_resolvery
Yee
03/09/2023, 9:12 PM
what is melody?
b
Bradley Worley
03/09/2023, 9:14 PMI think it’s the “top-level” module that flyte is inferring, but it’s not a package that exists in any sense. In other words, the code snippet I showed is running from e.g.
my_repo.melody.sub.modulemy_repo/melody/sub/module.pyy
Yee
03/09/2023, 9:15 PM
do you have an init file in melody?
b
Bradley Worley
03/09/2023, 9:15 PMYes, placed there by
bazel runy
Yee
03/09/2023, 9:28 PM
any chance you could push a repro somewhere that’s independent i can play around with?
Yee
03/09/2023, 9:28 PM
would help with debugging quite a bit
b
Bradley Worley
03/09/2023, 9:29 PMUnlikely 😔
Bradley Worley
03/09/2023, 9:29 PMIs there something wrong with my approach?
y
Yee
03/09/2023, 9:29 PM
i don’t think so.
Yee
03/09/2023, 9:30 PM
it’s unclear what’s happening though
Yee
03/09/2023, 9:30 PM
python module loading is pretty complicated i’ve found, and we do some esoteric things in the ‘tracker’ class
Yee
03/09/2023, 9:31 PM
the non-imperative workflows you speak of…
Yee
03/09/2023, 9:31 PM
are they built in the same folder in the same bazel run?
Yee
03/09/2023, 9:32 PM
‘melody’ showing up is a bit strange, but having it only show up in imperative is even stranger
b
Bradley Worley
03/09/2023, 9:35 PMThey are built from other folders, and registered using fast registration
y
Yee
03/09/2023, 9:35 PM
the non-imperative ones?
b
Bradley Worley
03/09/2023, 9:35 PMCorrect
y
Yee
03/09/2023, 9:35 PM
can you drop a dummy regular workflow in that same python file and see if it has the same problem?
b
Bradley Worley
03/09/2023, 9:36 PMI’ve been able to run a regular
@dynamicBradley Worley
03/09/2023, 9:38 PMBasically by registering it first with
Copy code
pyflyte register --image same_image $REPO_ROOT/my_repo/melody/sub/module.pyremote.execute()Bradley Worley
03/10/2023, 5:14 AMOkay it seems this was an oddity of our registration context, in vs out of a docker container, apologies for the false alarm
151 Views