https://flyte.org
Join Slack
Hi everyone! :wave: I’m attempting to register an imperative workflow for remote execution, using a ...
# flyte-support
w
Hi everyone! 👋 I’m attempting to register an imperative workflow for remote execution, using a bazel build environment, and I’m seeing a failure I don’t completely understand (MWE and error in the thread)
The process I’m going through:
Copy code
from flytekit import Workflow
from flytekit.configuration import (
  Config,
  SerializationSettings,
)
from flytekit.remote import FlyteRemote

remote = FlyteRemote(
    config=Config.auto(config_file=MY_CONFIG_PATH),
    default_project=MY_PROJECT,
    default_domain=MY_DOMAIN,
    data_upload_location=MY_S3_BUCKET,
)

workflow = Workflow(f"my.workflows.{wf_name}")
# build out the workflow ...

ss = SerializationSettings.for_image(
    image=f"{DOCKER_HOST}/flyte_workflow:latest",
    version=version_str,
    project=MY_PROJECT,
    domain=MY_DOMAIN,
)

remote.register_workflow(
    workflow,
    serialization_settings=ss,
)

execution = remote.execute(
    workflow,
    inputs=inputs,
    wait=False,
    project=MY_PROJECT,
    domain=MY_DOMAIN,
)
And the error:
Copy code
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f52a73cdf369d4ded82b-n0-0] terminated with exit code (1). Reason [Error]. Message: 
ask_cmd
    _execute_task(
  File "/app/pip_pypi__flytekit/flytekit/exceptions/scopes.py", line 160, in system_entry_point
    return wrapped(*args, **kwargs)
...
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'melody'
It seems like the issue is that 
pyflyte-execute
is getting a different 
--resolver
in the failed run:
Copy code
melody.pip_pypi__flytekit.flytekit.core.python_auto_container.default_task_resolver
Successful runs (that do not use imperative workflows) show this resolver:
Copy code
app.pip_pypi__flytekit.flytekit.core.python_auto_container.default_task_resolver
t
what is melody?
w
I think it’s the “top-level” module that flyte is inferring, but it’s not a package that exists in any sense. In other words, the code snippet I showed is running from e.g. 
my_repo.melody.sub.module
(
my_repo/melody/sub/module.py
)
t
do you have an init file in melody?
w
Yes, placed there by 
bazel run
t
any chance you could push a repro somewhere that’s independent i can play around with?
would help with debugging quite a bit
w
Unlikely 😔
Is there something wrong with my approach?
t
i don’t think so.
it’s unclear what’s happening though
python module loading is pretty complicated i’ve found, and we do some esoteric things in the ‘tracker’ class
the non-imperative workflows you speak of…
are they built in the same folder in the same bazel run?
‘melody’ showing up is a bit strange, but having it only show up in imperative is even stranger
w
They are built from other folders, and registered using fast registration
t
the non-imperative ones?
w
Correct
t
can you drop a dummy regular workflow in that same python file and see if it has the same problem?
w
I’ve been able to run a regular 
@dynamic
from that file
Basically by registering it first with
Copy code
pyflyte register --image same_image $REPO_ROOT/my_repo/melody/sub/module.py
And then calling 
remote.execute()
similarly
Okay it seems this was an oddity of our registration context, in vs out of a docker container, apologies for the false alarm
👌 1
153 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by