Thread
#announcements
    n

    Nada Saiyed

    1 month ago
    Hello, i am extending flytekit to create a custom plugin and i want to define some custom code when the pod gets terminated. are there any handlers that needs to be defined? 🧵
    I tried defining a basic
    SIGTERM
    handler, but that doesn’t get called. In fact, i see this error msg in properller’s log
    No plugin found for Handler-type...
    is there a specific way these handlers need to be defined?
    Kevin Su

    Kevin Su

    1 month ago
    you have to update propeller config map. check https://docs.flyte.org/projects/cookbook/en/latest/integrations.html#external-service-backend-plugins
    tasks:
      task-plugins:
        enabled-plugins:
          - container
          - sidecar
          - k8s-array
        default-for-task-types:
          container: container
          sidecar: sidecar
          container_array: k8s-array
          <task_type>: <your_handler> (by default, use container)
    n

    Nada Saiyed

    1 month ago
    What happens if I let it use the default?
    Kevin Su

    Kevin Su

    1 month ago
    propeller will launch a pod and run pyflyte-execute. you could also override the
    get_command
    in your custom plugin.
    basically we use container handler for regular
    @task
    n

    Nada Saiyed

    1 month ago
    right. i think i am okay with it using the default command. what i want to customize is its shutdown. how can i do that?
    Kevin Su

    Kevin Su

    1 month ago
    sorry, could you say more about exit strategy? what do you want to add in the plugin?
    n

    Nada Saiyed

    1 month ago
    i have a custom task plugin which creates an AWS resource. what i want is, when i execute this task as part of workflow, and i terminate the workflow, the AWS resource should be deleted. i would assume that when a workflow is aborted a
    SIGTERM
    is sent to the pod running that task (lmk if thats not the case?) and i am trying to define a shutdown hook which will run some cleanup code.
    i am not receiving a
    SIGTERM
    or a
    SIGINT
    in my code. how can i attach a shutdown hook that is run when the workflow is terminated
    Haytham Abuelfutuh

    Haytham Abuelfutuh

    1 month ago
    SIGTERM should be sent to the container: https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace Some reasons for why you may not be catching it: https://stackoverflow.com/questions/72544506/sigterm-not-sent-on-pod-delete It’s probably easier to just do a
    docker run
    with the same command that you can copy from the flyte Pod spec to try things locally… I would check the running processes in the container to see what’s process 1… etc. is it the shell or is it the python process..
    Ketan (kumare3)

    Ketan (kumare3)

    1 month ago
    @Nada Saiyed / @Haytham Abuelfutuh this is a fantastic addition to the plugin api - onTerminate - cc @Yee
    a

    Andrew Achkar

    1 month ago
    +1 as a plugin author having a place to define some cleanup/teardown actions would be really nice
    n

    Nada Saiyed

    1 month ago
    @Haytham Abuelfutuh yeaah.. the shell commands is PID 1. hmm, so the
    SIGTERM
    signal is sent to the the
    pyflyte_execute
    and its not forwarded to the python process.
    how should i run some piece of code at abrupt termination?
    Haytham Abuelfutuh

    Haytham Abuelfutuh

    1 month ago
    Can you share a screenshot for
    ps
    or whatever command you ran?
    n

    Nada Saiyed

    1 month ago
    Ketan (kumare3)

    Ketan (kumare3)

    1 month ago
    @Andrew Achkar can you file this issue?
    we will implement it as part of the next releae
    a

    Andrew Achkar

    1 month ago