acoustic-carpenter-78188
07/11/2023, 4:04 PMrecord_outputs. This is a problem when we try to use the output of a notebook task, e.g. as the return output of the workflow
"""An example of how to use Jupyter Notebooks in your workflows."""
from pathlib import Path
from typing import Tuple
from flytekit import workflow, kwtypes
from flytekit.types.file import FlyteFile
from flytekit.types.structured import StructuredDataset
from flytekitplugins.papermill import NotebookTask
from workflows.example_00_intro import get_data
data_analysis_task = NotebookTask(
name="workflows.data_analysis_task",
notebook_path=str(
Path(__file__).parent.parent.absolute()
/ "notebooks"
/ "data_analysis_task.ipynb"
),
render_deck=True,
disable_deck=False,
inputs=kwtypes(structured_dataset=StructuredDataset),
outputs=kwtypes(success=bool),
)
@workflow
def data_analysis_wf() -> Tuple[bool, FlyteFile, FlyteFile]:
data = get_data()
return data_analysis_task(structured_dataset=data)
Trying to pyflyte register this workflow will result on the following error:
RPC Failed, with Status: StatusCode.INTERNAL
details: failed to compile workflow for [resource_type:WORKFLOW project:"flytesnacks" domain:"development" name:"workflows.example_notebook_tasks.data_analysis_wf" version:"uMWWUPTSJ7aW7oSK79MRDg==" ] with err failed to compile workflow with err Collected Errors: 4
Error 0: Code: MismatchingTypes, Node Id: end-node, Description: Variable [out_nb] (type [blob:<format:"ipynb" > ]) doesn't match expected type [blob:<> ].
Error 1: Code: MismatchingTypes, Node Id: end-node, Description: Variable [out_rendered_nb] (type [blob:<format:"html" > ]) doesn't match expected type [blob:<> ].
Error 2: Code: ParameterNotBound, Node Id: end-node, Description: Parameter not bound [o1].
Error 3: Code: ParameterNotBound, Node Id: end-node, Description: Parameter not bound [o2].
On the other hand, changing the signature to the one explicit output:
def data_analysis_wf() -> bool:
We get this error:
Failed with Unknown Exception <class 'AssertionError'> Reason: The Workflow specification indicates only one return value, received 3
Expected behavior
Flyte should gracefully handle NotebookTask outputs somehow.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyte