Channels
datahub-flyte
scipy-2023-workshop
flyte-school
flyte-build
flyte-users-berlin
scipy-2023-sprint
auth
flyte-bazel
large-language-models
contribute
great-content
in-flyte-conversations
flyte-on-gcp
show-and-tell
flyte-github
linkedin-flyte
random
deployment
feature-discussions
flytelab
flytekit-java
integrations
ray-on-flyte
conference-talks
release
flyte-ui-ux
workflow-building-ui-proj
writing-w-sfloris
jobs
hacktoberfest-2022
torch-elastic
flyte-console
engineeringlabs
helsing-flyte
flyte-documentation
konan-integration
databricks-integration
ray-integration
wg-gpu-types
flytekit
ecosystem-unionml
scipy-2022-sprint
announcements
ask-the-community
flyte-deployment
introductions
events
Powered by
#flytekit
Title
h
honnix
06/21/2022, 10:04 AM
A quick question. What makes it special that these type of hack is required
https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-papermill/flytekitplugins/papermill/task.py#L117-L129
? BTW the comments around
https://github.com/flyteorg/flytekit/blob/501fcaab37f4cd270b869844d59bf71873f841fd/flytekit/extras/tasks/shell.py#L138-L147
seems to be a copy pasta thing that we can clean up a bit.
k
Ketan (kumare3)
06/21/2022, 2:22 PM
@honnix
so the notebook task can be any task - if I would redo it I will create a new task base called wrapper tasks
The idea is to allow a new facade for any other task type - like for example spark
And allow notebooks to execute spark tasks
h
honnix
06/21/2022, 2:29 PM
Right, that makes sense! I was wondering why not generalizing that “hacky” part. Will that wrapper task fit
https://github.com/flyteorg/flytekit/blob/07299d0b0e8d172e33ad48d719f9014effb5d96b/plugins/flytekit-greatexpectations/flytekitplugins/great_expectations/task.py
as well? Why is there no hacky part?
k
Ketan (kumare3)
06/21/2022, 2:33 PM
Ya I think we should - up for a contribution 🤩
3 Views
Post