Hi all, I created an issue here before realizing there was a Slack. Any ideas as to why the Python Ray example (from the docs) registers its workflow just fine, but the Jupyter Notebook example doesn't find any entities? I'm probably missing something obvious so apologies if that's the case. I noticed that VS Code thinks there is a

\n

after `@workflow`(unsurprising as Jupyter Notebooks are typically run in the browser obviously), not sure if that could be causing the problem.

@Peter Klingelhofer, you cannot register workflows present in ipynb files. You can, however, use FlyteRemote to register the tasks and workflows. https://docs.flyte.org/projects/flytekit/en/latest/design/control_plane.html#registering-entities

Thank you for the quick response. I think I'm having trouble figuring out what my flyte_entity should be. Let's assume my project name is

repo

, and I have the

ray_example.ipynb

file in the

workflows

folder, and I'm trying to add the workflow to the

development

domain. I was adding a new separate cell to the bottom of the Jupyter Notebook

ray_example.ipynb

file like so:

from flytekit.remote import FlyteRemote
from flytekit.configuration import Config, SerializationSettings, ImageConfig

# Using image pushed to local registry at localhost:30000
img = ImageConfig.from_images(
    "localhost:30000/repo:latest", {"repo": "localhost:30000/repo:latest"}
)

# FlyteRemote object is the main entrypoint to API
remote = FlyteRemote(
    config=Config.for_sandbox(),
    default_project="repo",
    default_domain="development",
)

# Get Task
# flyte_task = remote.fetch_task(name="workflows.ray_example", version="v1")
flyte_task = remote.fetch_task(
    name="workflows.ray_example",
    version="v1",
    project="repo",
    domain="development",
)

flyte_task = remote.register_task(
    entity=flyte_task,
    serialization_settings=SerializationSettings(image_config=None),
    version="v2",
)

flyte_workflow = remote.register_workflow(
    entity=flyte_task,
    serialization_settings=SerializationSettings(image_config=None),
    version="v1",
)
flyte_launch_plan = remote.register_launch_plan(entity=flyte_task, version="v1")

Yet I still receive the

FlyteEntityNotExistException

. Apologies if the answer is obvious. Thank you again so much for any help/assistance you can provide!

I'm assuming you've not registered the flyte task yet. In that case, you needn't fetch the task. Directly register it. Check out https://docs.flyte.org/projects/cookbook/en/latest/auto/case_studies/feature_engineering/feast_integration/Feast_Flyte_Demo.html example.

Thank you for your response @Samhita Alla. I believe in my code snippet above, that's what I've done in this section:

flyte_task = remote.register_task(
    entity=flyte_task,
    serialization_settings=SerializationSettings(image_config=None),
    version="v2",
)

Interestingly, this user suggests that registering workflows inside Jupyter notebooks is not possible: https://github.com/flyteorg/flyte/issues/3588#issuecomment-1509599891 If it is indeed possible, I would be happy to work on an MR to add an example Jupyter Notebook file to the Ray example (https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/ray_example/ray_example.html), just need to figure out how to get an example workflow working via a Jupyter Notebook. I'm just pushing the Docker image to the local registry at

localhost:30000

, which is what I would think would be the simplest implementation possible to run a workflow.

Papermill is for running jupyter notebook as a flyte task. In your case, I assume you're trying to register tasks and workflows that are present within your jupyter notebook which is absolutely possible. What @Kevin Su's telling is that you cannot register code present in your Jupyter with

pyflyte run

or

pyflyte register

. You need to use FlyteRemote to register your code. Can you try registering by following the example I've sent earlier?

Thanks so much for the help @Samhita Alla. I closed my GitHub issue as I did get the workflow to successfully register via importing the Jupyter Notebook via Papermill. However I'm still curious about FlyteRemote, I set up the FlyteRemote syntax, but I see that you said here that you can't use

pyflyte run

or

pyflyte register

, and I don't really see in the documentation regarding FlyteRemote what the equivalent commands would be to register workflows via FlyteRemote would be. If I'm using FlyteRemote, what command would need to be run to register workflows, since we can't use

pyflyte register

? Apologies again for the confusion on my part.

No problem! You'd have to use

register_task

/

register_workflow

/

register_launch_plan

/

register_script

function. FlyteRemote is a Python API. You can use it to programmatically register your code. https://github.com/flyteorg/flytekit/blob/e865db57d3bfbb7fb997417b052a05bc871cb0ed/flytekit/remote/remote.py

Ah thank you! I see now. That makes sense. 🙂