https://flyte.org logo
#contribute
Title
# contribute
n

Niels Bantilan

07/20/2023, 9:02 PM
👋 hi all! have you ever wanted to run flytekit code directly from within a jupyter notebook? Then I have the RFC for you! https://github.com/flyteorg/flyte/discussions/3878 Please add your comments to the RFC incubator post with ideas on what the ideal UX would be in your day-to-day work in jupyter notebooks.
Here’s a working example that can be found in the PR referenced in that post:
The notebook is effectively treated as a full flytekit script, and the
%notebook_run
magic essentially exports the code to a python file and runs it via
pyflyte run
.
This is probably not the ideal UX, so please add your ideas!
k

Ketan (kumare3)

07/21/2023, 1:24 AM
This is awesome
Cc @Mick Jermsurawong / @Pradithya Aria Pura
p

Pradithya Aria Pura

07/21/2023, 1:31 AM
@Lee Ning Jie Leon ^ cool stuff here, will improve our onboarding experience quite a lot.
m

Mick Jermsurawong

07/21/2023, 9:30 PM
oh nice! yah this will be super helpful to for quick iteration. So the idea here is that you prototype quickly on local notebook and get interactive feedback.. once you are happy and want to scale up you can luanch on cluster.
this is helpful when workflow lives in a self-contained code.. some notebooking system (ours, DB) have a notebook living as if a file in directory which has reference to local modules. So this change could be next step after treating notebook as "full flyte script"
n

Niels Bantilan

07/21/2023, 11:24 PM
So the idea here is that you prototype quickly on local notebook and get interactive feedback
yeah, so the idea is that I can interleave code that runs locally in my notebook runtime and send it up to run on a Flyte cluster when I need to scale. My question would be: would it make sense to support registering/running tasks/workflows in cells that are tagged “flyte” (or something), then be able to run those on a Flyte cluster? Basically the UX would be E.g.
Copy code
# runs on the notebook server
print("hello world")
Copy code
{tags: "flyte"}
# this cell is special: it MUST NOT share state with regular cells

@task
def t1(x: int): ...

@workflow
def wf(x: int) -> int:
    return t1(x=x)
Copy code
%% notebook_run --config ... wf --x 5
notebook_run cell output:
Copy code
Running "wf" on <remote_execution_urL>/<execution_id>
Retrieve this execution with

remote = FlyteRemote(...)
execution = remote.fetch_execution(<exection_id>)
… local analysis continues