Niels Bantilan
07/20/2023, 9:02 PM%notebook_run
magic essentially exports the code to a python file and runs it via pyflyte run
.Ketan (kumare3)
Pradithya Aria Pura
07/21/2023, 1:31 AMMick Jermsurawong
07/21/2023, 9:30 PMNiels Bantilan
07/21/2023, 11:24 PMSo the idea here is that you prototype quickly on local notebook and get interactive feedbackyeah, so the idea is that I can interleave code that runs locally in my notebook runtime and send it up to run on a Flyte cluster when I need to scale. My question would be: would it make sense to support registering/running tasks/workflows in cells that are tagged “flyte” (or something), then be able to run those on a Flyte cluster? Basically the UX would be E.g.
# runs on the notebook server
print("hello world")
…
{tags: "flyte"}
# this cell is special: it MUST NOT share state with regular cells
@task
def t1(x: int): ...
@workflow
def wf(x: int) -> int:
return t1(x=x)
…
%% notebook_run --config ... wf --x 5
notebook_run cell output:
Running "wf" on <remote_execution_urL>/<execution_id>
Retrieve this execution with
remote = FlyteRemote(...)
execution = remote.fetch_execution(<exection_id>)
…
local analysis continues