A quick question. What makes it special that these...
# flytekith
honnix
06/21/2022, 10:04 AMA quick question. What makes it special that these type of hack is required https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-papermill/flytekitplugins/papermill/task.py#L117-L129? BTW the comments around https://github.com/flyteorg/flytekit/blob/501fcaab37f4cd270b869844d59bf71873f841fd/flytekit/extras/tasks/shell.py#L138-L147 seems to be a copy pasta thing that we can clean up a bit.
k
Ketan (kumare3)
06/21/2022, 2:22 PM
@honnix so the notebook task can be any task - if I would redo it I will create a new task base called wrapper tasks
Ketan (kumare3)
06/21/2022, 2:23 PM
The idea is to allow a new facade for any other task type - like for example spark
Ketan (kumare3)
06/21/2022, 2:23 PM
And allow notebooks to execute spark tasks
h
honnix
06/21/2022, 2:29 PMRight, that makes sense! I was wondering why not generalizing that “hacky” part. Will that wrapper task fit https://github.com/flyteorg/flytekit/blob/07299d0b0e8d172e33ad48d719f9014effb5d96b/plugins/flytekit-greatexpectations/flytekitplugins/great_expectations/task.py as well? Why is there no hacky part?
k
Ketan (kumare3)
06/21/2022, 2:33 PM
Ya I think we should - up for a contribution 🤩
163 Views