Hi! I'm seeing a very weird error when trying to u...
# flyte-support
m
Hi! I'm seeing a very weird error when trying to use two different notebook tasks in a pipeline. It looks like only one is properly resolved, as throws an error regarding inputs not adhering to the types for the first task. Did anybody have the same issues?
h
@millions-addition-67809, can you give us more details? Maybe a repro?
m
Copy code
from flytekit import kwtypes, workflow, dynamic
from flytekitplugins.papermill import NotebookTask
from flytekit.types.file import HTMLPage

from flytekit import ImageSpec
from src.images import BaseImageSpec, NotebookImageSpec

task1 = NotebookTask(
    name="notebook_task1",
    notebook_path=str(
        pathlib.Path(__file__)
        .parent.absolute()
        .joinpath("notebooks/task1.ipynb")
    ),
    render_deck=True,
    enable_deck=True,
    inputs=kwtypes(
        some_input=str
    ),
    container_image=NotebookImageSpec,
)


task2 = NotebookTask(
    name="notebook_task2",
    notebook_path=str(
        pathlib.Path(__file__)
        .parent.absolute()
        .joinpath("notebooks/task2.ipynb")
    ),
    render_deck=True,
    enable_deck=True,
    inputs=kwtypes(
        some_other_input=str,
    ),
    container_image=NotebookImageSpec,
)


@dynamic(container_image=BaseImageSpec)
def run_dynamic_workflow() -> HTMLPage:
    _, nb2 = task2(some_other_input="some_input")
    return nb2


@workflow
def run_workflow() -> None:
    _, nb1 = task1(some_input="some_input")
    nb2 = run_dynamic_workflow()
when running on the remote cluster, the notebook task executed inside the dynamic workflow is actually task1:
Zrzut ekranu 2024-08-1 o 09.40.43.png
Hi guys, anyone knows the solution?
h
@millions-addition-67809, I'll be focusing on this today. Expect an update in the next couple of hours.
@millions-addition-67809, I confirmed the issue. This has been an issue since the version 1.10.0 of the papermill plugin. https://github.com/flyteorg/flytekit/pull/2660/ is the fix.
@millions-addition-67809, flytekitplugins-papermill 1.13.3 is out and it contains a fix for the issue you're seeing. Can you give it a try?