Hello, i am extending flytekit to create a custom ...
# announcements
p
Hello, i am extending flytekit to create a custom plugin and i want to define some custom code when the pod gets terminated. are there any handlers that needs to be defined? 🧵
I tried defining a basic
SIGTERM
handler, but that doesn’t get called. In fact, i see this error msg in properller’s log
Copy code
No plugin found for Handler-type...
is there a specific way these handlers need to be defined?
g
you have to update propeller config map. check https://docs.flyte.org/projects/cookbook/en/latest/integrations.html#external-service-backend-plugins
Copy code
tasks:
  task-plugins:
    enabled-plugins:
      - container
      - sidecar
      - k8s-array
    default-for-task-types:
      container: container
      sidecar: sidecar
      container_array: k8s-array
      <task_type>: <your_handler> (by default, use container)
p
What happens if I let it use the default?
g
propeller will launch a pod and run pyflyte-execute. you could also override the
get_command
in your custom plugin.
basically we use container handler for regular
@task
p
right. i think i am okay with it using the default command. what i want to customize is its shutdown. how can i do that?
g
sorry, could you say more about exit strategy? what do you want to add in the plugin?
p
i have a custom task plugin which creates an AWS resource. what i want is, when i execute this task as part of workflow, and i terminate the workflow, the AWS resource should be deleted. i would assume that when a workflow is aborted a
SIGTERM
is sent to the pod running that task (lmk if thats not the case?) and i am trying to define a shutdown hook which will run some cleanup code.
i am not receiving a
SIGTERM
or a
SIGINT
in my code. how can i attach a shutdown hook that is run when the workflow is terminated
h
SIGTERM should be sent to the container: https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace Some reasons for why you may not be catching it: https://stackoverflow.com/questions/72544506/sigterm-not-sent-on-pod-delete It’s probably easier to just do a
docker run
with the same command that you can copy from the flyte Pod spec to try things locally… I would check the running processes in the container to see what’s process 1… etc. is it the shell or is it the python process..
f
@plain-carpenter-67621 / @high-park-82026 this is a fantastic addition to the plugin api - onTerminate - cc @thankful-minister-83577
t
+1 as a plugin author having a place to define some cleanup/teardown actions would be really nice
👍 1
p
@high-park-82026 yeaah.. the shell commands is PID 1. hmm, so the
SIGTERM
signal is sent to the the
pyflyte_execute
and its not forwarded to the python process.
how should i run some piece of code at abrupt termination?
h
Can you share a screenshot for
ps
or whatever command you ran?
p
message has been deleted
f
@thankful-dress-89577 can you file this issue?
we will implement it as part of the next releae
❤️ 3
👍 1
t
171 Views