acoustic-carpenter-78188
06/27/2023, 10:37 AMimport os
import pathlib
from flytekitplugins.papermill import NotebookTask
from flytekitplugins.awsbatch import AWSBatchConfig
config = AWSBatchConfig(
platformCapabilities="EC2",
)
backtest_wages = NotebookTask(
name="test-notebook",
notebook_path=os.path.join(
pathlib.Path(__file__).parent.parent.absolute(), "notebooks/test-notebook.ipynb"
),
render_deck=True,
task_config=config
)
Tried to use AWS batch job to run the notebook task using the code example above, got an error:
logger.debug(f"Starting _dispatch_execute for {task_def.name}")
AttributeError: 'function' object has no attribute 'name'
When I looked at the batch job docker command:
["pyflyte-execute","--inputs","<s3://fiolito-eks-cluster/metadata/propeller/liquidity-forecast-development-alg58fr86fmkgh9nqnf4/n0/data/inputs.pb>","--output-prefix","<s3://fiolito-eks-cluster/metadata/propeller/liquidity-forecast-development-alg58fr86fmkgh9nqnf4/n0/data/0/0>","--raw-output-data-prefix","<s3://fiolito-eks-cluster/99/alg58fr86fmkgh9nqnf4-n0-0>","--resolver","flytekit.core.python_auto_container.default_task_resolver","--","task-module","flytekitplugins.papermill.task","task-name","_dummy_task_func"]
It seems that the _dummy_task_func is wrong which should be test-notebook and it comes from:
https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-papermill/flytekitplugins/papermill/task.py#L145
Expected behavior
Notebook task can run as an AWS Map Task without throwing exception.
Additional context to reproduce
Python3.10
flytekit 1.7.0
flytekitplugin-awsbatch 1.7.0
flytekitplugin-papermil 1.7.0
Screenshots
image▾
image▾
image▾