strong-oyster-56522
08/23/2022, 9:01 PMsidecar
task and the pod can be terminated as expected using the pod plugin. However, when running a workflow with a notebook task for papermill, the _task_type_ changes to nb-sidecar
. After that, the pod for this task can not be terminated and I don't know why. It seems like the same issue when we launch a workflow without the proper settings for pod plugin.
A code snipet for the task:
def generate_por_spec_for_task():
primary_container = V1Container(name="primary")
pod_spec = V1PodSpec(containers=[primary_container])
return pod_spec
nb = NotebookTask(
name="simple-nb",
task_config=Pod(pod_spec=generate_por_spec_for_task(), primary_container_name="primary"),
notebook_path=os.path.join(
pathlib.Path(file).parent.absolute(), "nb-simple.ipynb"
),
inputs=kwtypes(v=float),
outputs=kwtypes(square=float),
)
A log message in flytepropeller pod is the following:
{"json":{"exec_id":"ahkm5jd76h8l2gn46zt6","node":"n1","ns":"flyteexamples-development","res_ver":"209788243","routine":"worker-2","tasktype":"nb-sidecar","wf":"flyteexamples:development:flyte.workflows.simple.nb_to_python_wf"},"level":"warning","msg":"No plugin found for Handler-type [nb-sidecar], defaulting to [container]","ts":"2022-edited"}
Does anyone has insights about what could possibly going on?glamorous-carpet-83516
08/24/2022, 5:40 AMstrong-oyster-56522
08/24/2022, 12:47 PMtasks:
task-plugins:
default-for-task-types:
container: container
container_array: k8s-array
sidecar: sidecar
enabled-plugins:
- container
- sidecar
- k8s-array
There is nothing specific for nb-task or papermill plugin...glamorous-carpet-83516
08/24/2022, 2:57 PMshy-holiday-15500
08/24/2022, 3:10 PMshy-holiday-15500
08/24/2022, 3:13 PMshy-holiday-15500
08/24/2022, 3:20 PMNo plugin found for Handler-type [nb-python-task], defaulting to [container]
, because the nb tasks just use the default container plugin, they don't need any special plugin in flyte propeller (all nb execution code lives in flytekit).
I wonder if it needs to default to sidecar
in this case instead of just container
? Happy to investigate that line of thought more later today, or w/ some more info on the observed behaviourstrong-oyster-56522
08/24/2022, 4:53 PMsidecar
task. It didn't work. But I think that it just don't work like that. I don't know how to change the default task to sidecar
instead of container
... Maybe worth trying that to see what happens...shy-holiday-15500
08/24/2022, 5:39 PMsidecar
task type all that much, but presumably propeller must have some logic for those to tear down the sidecar that isn't getting used here. I suspect there is a bug here and we need to tweak something so that propeller uses a sidecar-aware plugin, and happy to help continue debugging, but will be slower since I'm new to this part of the code.
I can think of an inelegant workaround if you need this quickly: you could use a plain python task and use the python papermill library in it to run your notebook.strong-oyster-56522
08/24/2022, 6:44 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
glamorous-carpet-83516
08/25/2022, 2:11 PMnb-
from task typefreezing-airport-6809
strong-oyster-56522
08/25/2022, 8:42 PMfreezing-airport-6809
strong-oyster-56522
08/29/2022, 8:56 PMglamorous-carpet-83516
08/31/2022, 9:40 AMpip install git+<https://github.com/flyteorg/flytekit@papermill-bug>
https://github.com/flyteorg/flytekit/pull/1143