Hi, We have enabled the following two plugins in Flyte:

k8s:
  # inject-finalizer: Prevent GKE garbage collector from deleting the Pod before Propeller observes it
  inject-finalizer: true
  # delete-resource-on-finalize: After Propeller observes the Pod, remove the Pod to avoid dangling resources
  delete-resource-on-finalize: true

However, we are still encountering an issue where completed pods remain in the cluster for a long time. At the same time, the corresponding tasks in Flyte continue to show as "running." It also appears that Flyte Propeller is processing tasks very slowly. This issue occurs only intermittently (not consistently). Could someone help us understand why this might be happening and which configurations or components we should inspect to debug this issue?

your kubeclient config is probably incorrectly configured in propeller

This is the propeller config we have

ohh your kubeconfig is high

Can you give any suggestions, how can i fix this issue ?

did you see any error in the flytepropeller pod?

flytepropeller logs are very huge , can you specify the keyword that should be checked in logs ?

containerStatus IndexOutOfBound, requested [0], but total containerStatuses [0] in pod phase [Pending] Failed to record taskEvent, error [AlreadyExists: Event already exists, caused by [rpc error: code = AlreadyExists desc = have already recorded task execution phase QUEUED (version: 1) I can see above two errors majorly, first one is because the task pod is pending due to node unavailability, but i dont think its the problem. But not sure about second error