Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hello, i am extending flytekit to create a custom plugin and i want to define some custom code when the pod gets terminated. are there any handlers that needs to be defined? :thread:

I tried defining a basic `SIGTERM` handler, but that doesn’t get called.

In fact, i see this error msg in properller’s log
```No plugin found for Handler-type...```
is there a specific way these handlers need to be defined?

you have to update propeller config map. check <https://docs.flyte.org/projects/cookbook/en/latest/integrations.html#external-service-backend-plugins>

```tasks:
  task-plugins:
    enabled-plugins:
      - container
      - sidecar
      - k8s-array
    default-for-task-types:
      container: container
      sidecar: sidecar
      container_array: k8s-array
      &lt;task_type&gt;: &lt;your_handler&gt; (by default, use container)```


What happens if I let it use the default?

propeller will launch a pod and run <https://github.com/flyteorg/flytekit/blob/ffd942c95049906db1abc8f217584495b50b17c9/flytekit/core/python_auto_container.py#L112-L132|pyflyte-execute>. you could also override the `get_command` in your custom plugin.

basically we use container handler for regular `@task`

right. i think i am okay with it using the default command. what i want to customize is its shutdown. how can i do that?

sorry, could you say more about exit strategy? what do you want to add in the plugin?

i have a custom task plugin which creates an AWS resource. what i want is, when i execute this task as part of workflow, and i terminate the workflow, the AWS resource should be deleted. i would assume that when a workflow is aborted a `SIGTERM` is sent to the pod running that task (lmk if thats not the case?) and i am trying to define a shutdown hook which will run some cleanup code.

i am not receiving a `SIGTERM` or a `SIGINT` in my code. how can i attach a shutdown hook that is run when the workflow is terminated

SIGTERM should be sent to the container: <https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace>

Some reasons for why you may not be catching it: <https://stackoverflow.com/questions/72544506/sigterm-not-sent-on-pod-delete>

It’s probably easier to just do a `docker run` with the same command that you can copy from the flyte Pod spec to try things locally…
I would check the running processes in the container to see what’s process 1… etc. is it the shell or is it the python process..

<@U03KZJ7AXKR> / <@UNW4VP36V> this is a fantastic addition to the plugin api - onTerminate - cc <@UNR3C6Y4T> 

+1 as a plugin author having a place to define some cleanup/teardown actions would be really nice

<@UNW4VP36V> yeaah..  the shell commands is PID 1. hmm, so the `SIGTERM` signal is sent to the the `pyflyte_execute` and its not forwarded to the python process.

how should i run some piece of code at abrupt termination?

Can you share a screenshot for `ps` or whatever command you ran?

we will implement it as part of the next releae

<https://github.com/flyteorg/flyte/issues/2810>