Hello team. I am attempting to programmatically re...
# flyte-support
s
Hello team. I am attempting to programmatically register my workflows through
remote.register_script(...)
but I keep getting a
TaskReferenceNotFound
error saying that workflows.s3_ingest.initialize_resources was not found. Similarly, when using
remote.register_workflow(...)
I get an error on execution time that module workflows was not found. At the moment, I am attempting to run these workflows from a monorepo to utilize a local library imported with a relative import in poetry. Do I have to register each task on its own before registering the workflow with
remote.register_script()
?
Attempting to register the task before registering the workflow yielded no difference, and I can see the task in the ui. I'm guessing it has to do with the versions, but I am not sure how to move forward.
Same thing happens when attempting to register the example workflow from the dockerfile template when using
register_script()
. If I use
register_workflow()
, registration works, but when executing the workflow, I get the
ModuleNotFoundError
mentioned earlier.
t
are the workflows and the registration call present in the same file? could you move the workflows to an adjacent module and import them in the file where you're registering when using
register_workflow
?
s
They are not in the same file. My current structure looks something like this
Copy code
root
|->other projects
|->...
|->backend
   |->other python projects
   |->common library
   |->flyte-worker
      |->worker
      |  |->workflows
      |     |-><workflow files>
      |->main.py <-registered and executed here
For more context, I have managed to fast-register the workflows from the CLI with a custom image to include the library, and moved on to the actual content of the workflows. But I will need to circle back to the registration soon-ish.
t
this directory structure works for me:
Copy code
from flytekit.remote import FlyteRemote
from flytekit.configuration import Config, ImageConfig
from wf.test_json import jsons_iter_wf
from flytekit.configuration import SerializationSettings, FastSerializationSettings


remote = FlyteRemote(config=Config.auto(), default_project="flytesnacks", default_domain="development")

_, native_url = remote.fast_package(root=".")

flyte_workflow = remote.register_workflow(
    entity=jsons_iter_wf,
    serialization_settings=SerializationSettings(
        image_config=ImageConfig.auto(img_name="..."),
        project="flytesnacks",
        domain="development",
        version="v1",
        fast_serialization_settings=FastSerializationSettings(enabled=True, destination_dir=".", distribution_location=native_url),
    ),
)
^^^ register.py
s
I haven't made any progress with this issue, but I want to thank you for the help. When I tried your solution I was still getting a
ModuleNotFoundError
when executing the workflow. It's probably an error I made when setting up the package that I'm not seeing. I will circle back to this as I said previously, but for now I'm registering from the CLI
👍 2
p
@sparse-dream-26689 Any success solving this? I'm facing the same issue, but only with flytekit version
>v1.11