https://flyte.org logo
#ask-the-community
Title
# ask-the-community
b

Bradley Worley

03/09/2023, 6:44 PM
Hi everyone! 👋 I’m attempting to register an imperative workflow for remote execution, using a bazel build environment, and I’m seeing a failure I don’t completely understand (MWE and error in the thread)
The process I’m going through:
Copy code
from flytekit import Workflow
from flytekit.configuration import (
  Config,
  SerializationSettings,
)
from flytekit.remote import FlyteRemote

remote = FlyteRemote(
    config=Config.auto(config_file=MY_CONFIG_PATH),
    default_project=MY_PROJECT,
    default_domain=MY_DOMAIN,
    data_upload_location=MY_S3_BUCKET,
)

workflow = Workflow(f"my.workflows.{wf_name}")
# build out the workflow ...

ss = SerializationSettings.for_image(
    image=f"{DOCKER_HOST}/flyte_workflow:latest",
    version=version_str,
    project=MY_PROJECT,
    domain=MY_DOMAIN,
)

remote.register_workflow(
    workflow,
    serialization_settings=ss,
)

execution = remote.execute(
    workflow,
    inputs=inputs,
    wait=False,
    project=MY_PROJECT,
    domain=MY_DOMAIN,
)
And the error:
Copy code
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f52a73cdf369d4ded82b-n0-0] terminated with exit code (1). Reason [Error]. Message: 
ask_cmd
    _execute_task(
  File "/app/pip_pypi__flytekit/flytekit/exceptions/scopes.py", line 160, in system_entry_point
    return wrapped(*args, **kwargs)
...
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'melody'
It seems like the issue is that
pyflyte-execute
is getting a different
--resolver
in the failed run:
Copy code
melody.pip_pypi__flytekit.flytekit.core.python_auto_container.default_task_resolver
Successful runs (that do not use imperative workflows) show this resolver:
Copy code
app.pip_pypi__flytekit.flytekit.core.python_auto_container.default_task_resolver
y

Yee

03/09/2023, 9:12 PM
what is melody?
b

Bradley Worley

03/09/2023, 9:14 PM
I think it’s the “top-level” module that flyte is inferring, but it’s not a package that exists in any sense. In other words, the code snippet I showed is running from e.g.
my_repo.melody.sub.module
(
my_repo/melody/sub/module.py
)
y

Yee

03/09/2023, 9:15 PM
do you have an init file in melody?
b

Bradley Worley

03/09/2023, 9:15 PM
Yes, placed there by
bazel run
y

Yee

03/09/2023, 9:28 PM
any chance you could push a repro somewhere that’s independent i can play around with?
would help with debugging quite a bit
b

Bradley Worley

03/09/2023, 9:29 PM
Unlikely 😔
Is there something wrong with my approach?
y

Yee

03/09/2023, 9:29 PM
i don’t think so.
it’s unclear what’s happening though
python module loading is pretty complicated i’ve found, and we do some esoteric things in the ‘tracker’ class
the non-imperative workflows you speak of…
are they built in the same folder in the same bazel run?
‘melody’ showing up is a bit strange, but having it only show up in imperative is even stranger
b

Bradley Worley

03/09/2023, 9:35 PM
They are built from other folders, and registered using fast registration
y

Yee

03/09/2023, 9:35 PM
the non-imperative ones?
b

Bradley Worley

03/09/2023, 9:35 PM
Correct
y

Yee

03/09/2023, 9:35 PM
can you drop a dummy regular workflow in that same python file and see if it has the same problem?
b

Bradley Worley

03/09/2023, 9:36 PM
I’ve been able to run a regular
@dynamic
from that file
Basically by registering it first with
Copy code
pyflyte register --image same_image $REPO_ROOT/my_repo/melody/sub/module.py
And then calling
remote.execute()
similarly
Okay it seems this was an oddity of our registration context, in vs out of a docker container, apologies for the false alarm