acoustic-carpenter-78188
08/15/2023, 9:12 PM@flytekit.task(
task_config=Elastic(
nnodes=3,
nproc_per_node=8,
rdzv_configs={"timeout": 1800, "join_timeout": 1800}
),
)
Will create a task execution with 3 instances fulfilling the requested resources. However, when I terminate the task execution from UI, I can still see the pods running on k8s:
llm-fine-tuning-development f8d2779abac0140fea1e-n1-0-worker-0 ● 1/1 0 Running 10 ││
llm-fine-tuning-development f8d2779abac0140fea1e-n1-0-worker-1 ● 1/1 0 Running 10 ││
llm-fine-tuning-development f8d2779abac0140fea1e-n1-0-worker-2 ● 1/1 0 Running 10 │
One has to manually kill the pods.
Expected behavior
When I terminate this task, all the pods should be killed automatically
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyte