https://flyte.org logo
#ray-integration
Title
# ray-integration
p

Peter Klingelhofer

04/12/2023, 9:00 PM
Hi all, I created an issue here before realizing there was a Slack. Any ideas as to why the Python Ray example (from the docs) registers its workflow just fine, but the Jupyter Notebook example doesn't find any entities? I'm probably missing something obvious so apologies if that's the case. I noticed that VS Code thinks there is a
\n
after `@workflow`(unsurprising as Jupyter Notebooks are typically run in the browser obviously), not sure if that could be causing the problem.
s

Samhita Alla

04/13/2023, 6:24 AM
@Peter Klingelhofer, you cannot register workflows present in ipynb files. You can, however, use FlyteRemote to register the tasks and workflows. https://docs.flyte.org/projects/flytekit/en/latest/design/control_plane.html#registering-entities
You can include this code in a separate cell in your jupyter notebook and run it.
p

Peter Klingelhofer

04/13/2023, 12:14 PM
Thank you for the quick response. I think I'm having trouble figuring out what my flyte_entity should be. Let's assume my project name is
repo
, and I have the
ray_example.ipynb
file in the
workflows
folder, and I'm trying to add the workflow to the
development
domain. I was adding a new separate cell to the bottom of the Jupyter Notebook
ray_example.ipynb
file like so:
Copy code
from flytekit.remote import FlyteRemote
from flytekit.configuration import Config, SerializationSettings, ImageConfig

# Using image pushed to local registry at localhost:30000
img = ImageConfig.from_images(
    "localhost:30000/repo:latest", {"repo": "localhost:30000/repo:latest"}
)

# FlyteRemote object is the main entrypoint to API
remote = FlyteRemote(
    config=Config.for_sandbox(),
    default_project="repo",
    default_domain="development",
)

# Get Task
# flyte_task = remote.fetch_task(name="workflows.ray_example", version="v1")
flyte_task = remote.fetch_task(
    name="workflows.ray_example",
    version="v1",
    project="repo",
    domain="development",
)

flyte_task = remote.register_task(
    entity=flyte_task,
    serialization_settings=SerializationSettings(image_config=None),
    version="v2",
)

flyte_workflow = remote.register_workflow(
    entity=flyte_task,
    serialization_settings=SerializationSettings(image_config=None),
    version="v1",
)
flyte_launch_plan = remote.register_launch_plan(entity=flyte_task, version="v1")
Yet I still receive the
FlyteEntityNotExistException
. Apologies if the answer is obvious. Thank you again so much for any help/assistance you can provide!
s

Samhita Alla

04/14/2023, 10:08 AM
I'm assuming you've not registered the flyte task yet. In that case, you needn't fetch the task. Directly register it. Check out https://docs.flyte.org/projects/cookbook/en/latest/auto/case_studies/feature_engineering/feast_integration/Feast_Flyte_Demo.html example.
p

Peter Klingelhofer

04/15/2023, 1:11 PM
Thank you for your response @Samhita Alla. I believe in my code snippet above, that's what I've done in this section:
Copy code
flyte_task = remote.register_task(
    entity=flyte_task,
    serialization_settings=SerializationSettings(image_config=None),
    version="v2",
)
Interestingly, this user suggests that registering workflows inside Jupyter notebooks is not possible: https://github.com/flyteorg/flyte/issues/3588#issuecomment-1509599891 If it is indeed possible, I would be happy to work on an MR to add an example Jupyter Notebook file to the Ray example (https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/ray_example/ray_example.html), just need to figure out how to get an example workflow working via a Jupyter Notebook. I'm just pushing the Docker image to the local registry at
localhost:30000
, which is what I would think would be the simplest implementation possible to run a workflow.
s

Samhita Alla

04/17/2023, 4:55 AM
Papermill is for running jupyter notebook as a flyte task. In your case, I assume you're trying to register tasks and workflows that are present within your jupyter notebook which is absolutely possible. What @Kevin Su's telling is that you cannot register code present in your Jupyter with
pyflyte run
or
pyflyte register
. You need to use FlyteRemote to register your code. Can you try registering by following the example I've sent earlier?
p

Peter Klingelhofer

04/17/2023, 1:45 PM
Thanks so much for the help @Samhita Alla. I closed my GitHub issue as I did get the workflow to successfully register via importing the Jupyter Notebook via Papermill. However I'm still curious about FlyteRemote, I set up the FlyteRemote syntax, but I see that you said here that you can't use
pyflyte run
or
pyflyte register
, and I don't really see in the documentation regarding FlyteRemote what the equivalent commands would be to register workflows via FlyteRemote would be. If I'm using FlyteRemote, what command would need to be run to register workflows, since we can't use
pyflyte register
? Apologies again for the confusion on my part.
s

Samhita Alla

04/17/2023, 3:18 PM
No problem! You'd have to use
register_task
/
register_workflow
/
register_launch_plan
/
register_script
function. FlyteRemote is a Python API. You can use it to programmatically register your code. https://github.com/flyteorg/flytekit/blob/e865db57d3bfbb7fb997417b052a05bc871cb0ed/flytekit/remote/remote.py
p

Peter Klingelhofer

04/17/2023, 3:58 PM
Ah thank you! I see now. That makes sense. 🙂
36 Views