https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
b

Bradley Worley

03/09/2023, 6:44 PM
Hi everyone! šŸ‘‹ Iā€™m attempting to register an imperative workflow for remote execution, using a bazel build environment, and Iā€™m seeing a failure I donā€™t completely understand (MWE and error in the thread)
The process Iā€™m going through:
Copy code
from flytekit import Workflow
from flytekit.configuration import (
  Config,
  SerializationSettings,
)
from flytekit.remote import FlyteRemote

remote = FlyteRemote(
    config=Config.auto(config_file=MY_CONFIG_PATH),
    default_project=MY_PROJECT,
    default_domain=MY_DOMAIN,
    data_upload_location=MY_S3_BUCKET,
)

workflow = Workflow(f"my.workflows.{wf_name}")
# build out the workflow ...

ss = SerializationSettings.for_image(
    image=f"{DOCKER_HOST}/flyte_workflow:latest",
    version=version_str,
    project=MY_PROJECT,
    domain=MY_DOMAIN,
)

remote.register_workflow(
    workflow,
    serialization_settings=ss,
)

execution = remote.execute(
    workflow,
    inputs=inputs,
    wait=False,
    project=MY_PROJECT,
    domain=MY_DOMAIN,
)
And the error:
Copy code
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f52a73cdf369d4ded82b-n0-0] terminated with exit code (1). Reason [Error]. Message: 
ask_cmd
    _execute_task(
  File "/app/pip_pypi__flytekit/flytekit/exceptions/scopes.py", line 160, in system_entry_point
    return wrapped(*args, **kwargs)
...
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'melody'
It seems like the issue is that 
pyflyte-execute
is getting a different 
--resolver
in the failed run:
Copy code
melody.pip_pypi__flytekit.flytekit.core.python_auto_container.default_task_resolver
Successful runs (that do not use imperative workflows) show this resolver:
Copy code
app.pip_pypi__flytekit.flytekit.core.python_auto_container.default_task_resolver
y

Yee

03/09/2023, 9:12 PM
what is melody?
b

Bradley Worley

03/09/2023, 9:14 PM
I think itā€™s the ā€œtop-levelā€ module that flyte is inferring, but itā€™s not a package that exists in any sense. In other words, the code snippet I showed is running from e.g. 
my_repo.melody.sub.module
(
my_repo/melody/sub/module.py
)
y

Yee

03/09/2023, 9:15 PM
do you have an init file in melody?
b

Bradley Worley

03/09/2023, 9:15 PM
Yes, placed there by 
bazel run
y

Yee

03/09/2023, 9:28 PM
any chance you could push a repro somewhere thatā€™s independent i can play around with?
would help with debugging quite a bit
b

Bradley Worley

03/09/2023, 9:29 PM
Unlikely šŸ˜”
Is there something wrong with my approach?
y

Yee

03/09/2023, 9:29 PM
i donā€™t think so.
itā€™s unclear whatā€™s happening though
python module loading is pretty complicated iā€™ve found, and we do some esoteric things in the ā€˜trackerā€™ class
the non-imperative workflows you speak ofā€¦
are they built in the same folder in the same bazel run?
ā€˜melodyā€™ showing up is a bit strange, but having it only show up in imperative is even stranger
b

Bradley Worley

03/09/2023, 9:35 PM
They are built from other folders, and registered using fast registration
y

Yee

03/09/2023, 9:35 PM
the non-imperative ones?
b

Bradley Worley

03/09/2023, 9:35 PM
Correct
y

Yee

03/09/2023, 9:35 PM
can you drop a dummy regular workflow in that same python file and see if it has the same problem?
b

Bradley Worley

03/09/2023, 9:36 PM
Iā€™ve been able to run a regular 
@dynamic
from that file
Basically by registering it first with
Copy code
pyflyte register --image same_image $REPO_ROOT/my_repo/melody/sub/module.py
And then calling 
remote.execute()
similarly
Okay it seems this was an oddity of our registration context, in vs out of a docker container, apologies for the false alarm
Powered by