Hello, i am extending flytekit to create a custom plugin and i want to define some custom code when the pod gets terminated. are there any handlers that needs to be defined? 🧵

you have to update propeller config map. check https://docs.flyte.org/projects/cookbook/en/latest/integrations.html#external-service-backend-plugins

tasks:
  task-plugins:
    enabled-plugins:
      - container
      - sidecar
      - k8s-array
    default-for-task-types:
      container: container
      sidecar: sidecar
      container_array: k8s-array
      <task_type>: <your_handler> (by default, use container)

What happens if I let it use the default?

propeller will launch a pod and run pyflyte-execute. you could also override the

get_command

in your custom plugin.

right. i think i am okay with it using the default command. what i want to customize is its shutdown. how can i do that?

sorry, could you say more about exit strategy? what do you want to add in the plugin?

i have a custom task plugin which creates an AWS resource. what i want is, when i execute this task as part of workflow, and i terminate the workflow, the AWS resource should be deleted. i would assume that when a workflow is aborted a

SIGTERM

is sent to the pod running that task (lmk if thats not the case?) and i am trying to define a shutdown hook which will run some cleanup code.

SIGTERM should be sent to the container: https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace Some reasons for why you may not be catching it: https://stackoverflow.com/questions/72544506/sigterm-not-sent-on-pod-delete It’s probably easier to just do a

docker run

with the same command that you can copy from the flyte Pod spec to try things locally… I would check the running processes in the container to see what’s process 1… etc. is it the shell or is it the python process..

@Nada Saiyed / @Haytham Abuelfutuh this is a fantastic addition to the plugin api - onTerminate - cc @Yee

+1 as a plugin author having a place to define some cleanup/teardown actions would be really nice

@Haytham Abuelfutuh yeaah.. the shell commands is PID 1. hmm, so the

SIGTERM

signal is sent to the the

pyflyte_execute

and its not forwarded to the python process.

Can you share a screenshot for

ps

or whatever command you ran?

@Andrew Achkar can you file this issue?