Channels
#contribute
Title
# contribute
n
Niels Bantilan
07/20/2023, 9:02 PMđź‘‹ hi all! have you ever wanted to run flytekit code directly from within a jupyter notebook?
Then I have the RFC for you! https://github.com/flyteorg/flyte/discussions/3878
Please add your comments to the RFC incubator post with ideas on what the ideal UX would be in your day-to-day work in jupyter notebooks.
Here’s a working example that can be found in the PR referenced in that post:
The notebook is effectively treated as a full flytekit script, and the
%notebook_runpyflyte runThis is probably not the ideal UX, so please add your ideas!
k
Ketan (kumare3)
07/21/2023, 1:24 AM
This is awesome
Cc @Mick Jermsurawong / @Pradithya Aria Pura
p
Pradithya Aria Pura
07/21/2023, 1:31 AM@Lee Ning Jie Leon ^ cool stuff here, will improve our onboarding experience quite a lot.
m
Mick Jermsurawong
07/21/2023, 9:30 PMoh nice! yah this will be super helpful to for quick iteration. So the idea here is that you prototype quickly on local notebook and get interactive feedback.. once you are happy and want to scale up you can luanch on cluster.
this is helpful when workflow lives in a self-contained code.. some notebooking system (ours, DB) have a notebook living as if a file in directory which has reference to local modules. So this change could be next step after treating notebook as "full flyte script"
n
Niels Bantilan
07/21/2023, 11:24 PMSo the idea here is that you prototype quickly on local notebook and get interactive feedbackyeah, so the idea is that I can interleave code that runs locally in my notebook runtime and send it up to run on a Flyte cluster when I need to scale. My question would be: would it make sense to support registering/running tasks/workflows in cells that are tagged “flyte” (or something), then be able to run those on a Flyte cluster? Basically the UX would be E.g.
Copy code
# runs on the notebook server
print("hello world") Copy code
{tags: "flyte"}
# this cell is special: it MUST NOT share state with regular cells
@task
def t1(x: int): ...
@workflow
def wf(x: int) -> int:
return t1(x=x) Copy code
%% notebook_run --config ... wf --x 5 Copy code
Running "wf" on <remote_execution_urL>/<execution_id>
Retrieve this execution with
remote = FlyteRemote(...)
execution = remote.fetch_execution(<exection_id>)