https://flyte.org logo
#announcements
Title
# announcements
n

Nada Saiyed

08/24/2022, 4:13 PM
Hello, i am extending flytekit to create a custom plugin and i want to define some custom code when the pod gets terminated. are there any handlers that needs to be defined? 🧵
I tried defining a basic
SIGTERM
handler, but that doesn’t get called. In fact, i see this error msg in properller’s log
Copy code
No plugin found for Handler-type...
is there a specific way these handlers need to be defined?
k

Kevin Su

08/24/2022, 4:23 PM
you have to update propeller config map. check https://docs.flyte.org/projects/cookbook/en/latest/integrations.html#external-service-backend-plugins
Copy code
tasks:
  task-plugins:
    enabled-plugins:
      - container
      - sidecar
      - k8s-array
    default-for-task-types:
      container: container
      sidecar: sidecar
      container_array: k8s-array
      <task_type>: <your_handler> (by default, use container)
n

Nada Saiyed

08/24/2022, 4:25 PM
What happens if I let it use the default?
k

Kevin Su

08/24/2022, 4:28 PM
propeller will launch a pod and run pyflyte-execute. you could also override the
get_command
in your custom plugin.
basically we use container handler for regular
@task
n

Nada Saiyed

08/24/2022, 4:31 PM
right. i think i am okay with it using the default command. what i want to customize is its shutdown. how can i do that?
k

Kevin Su

08/24/2022, 4:34 PM
sorry, could you say more about exit strategy? what do you want to add in the plugin?
n

Nada Saiyed

08/24/2022, 4:40 PM
i have a custom task plugin which creates an AWS resource. what i want is, when i execute this task as part of workflow, and i terminate the workflow, the AWS resource should be deleted. i would assume that when a workflow is aborted a
SIGTERM
is sent to the pod running that task (lmk if thats not the case?) and i am trying to define a shutdown hook which will run some cleanup code.
i am not receiving a
SIGTERM
or a
SIGINT
in my code. how can i attach a shutdown hook that is run when the workflow is terminated
h

Haytham Abuelfutuh

08/24/2022, 5:24 PM
SIGTERM should be sent to the container: https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace Some reasons for why you may not be catching it: https://stackoverflow.com/questions/72544506/sigterm-not-sent-on-pod-delete It’s probably easier to just do a
docker run
with the same command that you can copy from the flyte Pod spec to try things locally… I would check the running processes in the container to see what’s process 1… etc. is it the shell or is it the python process..
k

Ketan (kumare3)

08/24/2022, 6:01 PM
@Nada Saiyed / @Haytham Abuelfutuh this is a fantastic addition to the plugin api - onTerminate - cc @Yee
a

Andrew Achkar

08/24/2022, 6:38 PM
+1 as a plugin author having a place to define some cleanup/teardown actions would be really nice
👍 1
n

Nada Saiyed

08/24/2022, 6:56 PM
@Haytham Abuelfutuh yeaah.. the shell commands is PID 1. hmm, so the
SIGTERM
signal is sent to the the
pyflyte_execute
and its not forwarded to the python process.
how should i run some piece of code at abrupt termination?
h

Haytham Abuelfutuh

08/24/2022, 6:57 PM
Can you share a screenshot for
ps
or whatever command you ran?
n

Nada Saiyed

08/24/2022, 6:59 PM
message has been deleted
k

Ketan (kumare3)

08/25/2022, 5:51 AM
@Andrew Achkar can you file this issue?
we will implement it as part of the next releae
❤️ 3
👍 1
a

Andrew Achkar

08/25/2022, 1:29 PM
30 Views