I've been thinking about more monitoring capabilities from Jupyter notebooks. Has anyone thought about using ipywidgets to mimic some of the functionality in FlyteRemote/FlyteConsole?
03/14/2023, 5:07 AM
Can you elaborate a bit more on this, Greg? Would like to understand your use case.
03/14/2023, 6:44 AM
right now, we have a lot of workflows that run on Flyte and some are long running (minutes to hours). Most of our users are running these workflows from Jupyter notebooks, waiting for workflow to finish, and upon completion, fetching its output. As it stands, monitoring capabilities from notebook are limited (use FlyteRemote, or leave Jupyter to go to FlyteConsole, monitor from there, go back to Jupyter when it's finished).
Ideally a user could see what is happening on Flyte from within a notebook cell (this is where I think ipywidgets would be nice), rather than switch contexts.
I've also ran into issues using FlyteRemote/FlyteConsole when you have thousands of subtasks - quickly finding the task that failed can be a challenging feat. Hoping to solve that as well, but may be out of scope
03/14/2023, 1:34 PM
Gotcha, but it may not be of high priority today. How about you create an issue (a feature request precisely) and we can track it from there if there's interest within the community?