Hey team! In our current Flyte -> Spark setup, we use our internal infra to run Spark jobs. This infra is owned by another team. Currently there's no shutdown hook implemented ie, if user terminated Flyte job, the Spark job keeps running. Question: What is recommended way to implement shutdown of Spark job when Flyte wf is manually terminated? I can think of couple of ways: 1. have shutdown handler which stops the Spark job when Flyte wf is being terminated 2. other is to use Flyte Agent to terminate any orphaned Spark jobs

this is a common problem. the best way to solve is to write a native integration for Flyte using FlyteAgents framework

Thanks!