Join Slack
Powered by
A quick question. What makes it special that these...
# flytekit
s
steep-jackal-21573
06/21/2022, 10:04 AM
A quick question. What makes it special that these type of hack is required
https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-papermill/flytekitplugins/papermill/task.py#L117-L129
? BTW the comments around
https://github.com/flyteorg/flytekit/blob/501fcaab37f4cd270b869844d59bf71873f841fd/flytekit/extras/tasks/shell.py#L138-L147
seems to be a copy pasta thing that we can clean up a bit.
f
freezing-airport-6809
06/21/2022, 2:22 PM
@steep-jackal-21573
so the notebook task can be any task - if I would redo it I will create a new task base called wrapper tasks
freezing-airport-6809
06/21/2022, 2:23 PM
The idea is to allow a new facade for any other task type - like for example spark
freezing-airport-6809
06/21/2022, 2:23 PM
And allow notebooks to execute spark tasks
s
steep-jackal-21573
06/21/2022, 2:29 PM
Right, that makes sense! I was wondering why not generalizing that “hacky” part. Will that wrapper task fit
https://github.com/flyteorg/flytekit/blob/07299d0b0e8d172e33ad48d719f9014effb5d96b/plugins/flytekit-greatexpectations/flytekitplugins/great_expectations/task.py
as well? Why is there no hacky part?
f
freezing-airport-6809
06/21/2022, 2:33 PM
Ya I think we should - up for a contribution 🤩
163
Views
Open in Slack
Previous
Next