inject-finalizer will prevent K8s from claiming back the pod

@freezing-airport-6809 thanks for the

inject-finalizer

suggestion, I tried it out and my original issue seems solved, but now I’m having a backlog of pods that are stuck in Terminating (for 4+ hours so far) and still have the finalizer on them. have you seen this before?

Would love to dive deeper

Also, what version of FlytePropeller are you using?

Also on the pods is the deletion timestamp set?

the deletion timestamp is set on the pods, here’s an example:

apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      <http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: "false"
    creationTimestamp: "2022-04-19T07:30:49Z"
    deletionGracePeriodSeconds: 0
    deletionTimestamp: "2022-04-19T10:36:03Z"
    finalizers:
    - flyte/flytek8s
    labels:
      domain: main
      execution-id: f11qb62y-n1-0-dn0-0
      interruptible: "true"
      node-id: n2
      project: sunflower
      shard-key: "4"
      task-name: sunflower-workflows-flyte-workflows-trim-fastqs-and-align-workf
      workflow-name: sunflower-workflows-flyte-workflows-trim-fastqs-and-align-workf
    name: f11qb62y-n1-0-dn0-0-n2-0
    namespace: sunflower-main
    ownerReferences:
    - apiVersion: <http://flyte.lyft.com/v1alpha1|flyte.lyft.com/v1alpha1>
      blockOwnerDeletion: true
      controller: true
      kind: flyteworkflow
      name: f11qb62y-n1-0-dn0-0
      uid: e0525cc8-17c2-463b-a5dd-342776c69bbd
    resourceVersion: "266395947"
    uid: 393c4d2b-42ad-4061-8c95-a5d4f6eabb67
  spec:
    affinity:
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - preference:
            matchExpressions:
            - key: <http://cloud.google.com/gke-preemptible|cloud.google.com/gke-preemptible>
              operator: Exists
          weight: 1
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: <http://k8s.freenome.net/node-role|k8s.freenome.net/node-role>
              operator: In
              values:
              - flyte-worker
            - key: <http://cloud.google.com/gke-preemptible|cloud.google.com/gke-preemptible>
              operator: In
              values:
              - "true"
    containers:
    - args:
      - pyflyte-execute
      - --inputs
      - <gs://freenome-orchid-staging-flyte-data/metadata/propeller/sunflower-main-f11qb62y-n1-0-dn0-0/n2/data/inputs.pb>
      - --output-prefix
      - <gs://freenome-orchid-staging-flyte-data/metadata/propeller/sunflower-main-f11qb62y-n1-0-dn0-0/n2/data/0>
      - --raw-output-data-prefix
      - <gs://freenome-orchid-staging-flyte-data/ak/f11qb62y-n1-0-dn0-0-n2-0>
      - --resolver
      - flytekit.core.python_auto_container.default_task_resolver
      - --
      - task-module
      - sunflower.workflows.flyte_workflows.trim_fastqs_and_align_workflow
      - task-name
      - align
      env:
      - name: FLYTE_INTERNAL_CONFIGURATION_PATH
        value: /usr/src/app/sunflower/workflows/config/workflows.config
      - name: FLYTE_INTERNAL_IMAGE
        value: <http://gcr.io/freenome-build/ap/sunflower:20220414.2|gcr.io/freenome-build/ap/sunflower:20220414.2>
      - name: FLYTE_INTERNAL_EXECUTION_WORKFLOW
        value: sunflower:main:sunflower.workflows.flyte_workflows.trim_fastqs_and_align_workflow.trim_fastqs_and_align_wf
      - name: FLYTE_INTERNAL_EXECUTION_ID
        value: f11qb62y-n1-0-dn0-0
      - name: FLYTE_INTERNAL_EXECUTION_PROJECT
        value: sunflower
      - name: FLYTE_INTERNAL_EXECUTION_DOMAIN
        value: main
      - name: FLYTE_ATTEMPT_NUMBER
        value: "0"
      - name: FLYTE_INTERNAL_TASK_PROJECT
        value: sunflower
      - name: FLYTE_INTERNAL_TASK_DOMAIN
        value: main
      - name: FLYTE_INTERNAL_TASK_NAME
        value: sunflower.workflows.flyte_workflows.trim_fastqs_and_align_workflow.align
      - name: FLYTE_INTERNAL_TASK_VERSION
        value: "20220414.2"
      - name: FLYTE_INTERNAL_PROJECT
        value: sunflower
      - name: FLYTE_INTERNAL_DOMAIN
        value: main
      - name: FLYTE_INTERNAL_NAME
        value: sunflower.workflows.flyte_workflows.trim_fastqs_and_align_workflow.align
      - name: FLYTE_INTERNAL_VERSION
        value: "20220414.2"
      - name: SUNFLOWER_STATIC_DATA_GCS_PATH
        value: <gs://freenome-orchid-staging-static-data>
      image: <http://gcr.io/freenome-build/ap/sunflower:20220414.2|gcr.io/freenome-build/ap/sunflower:20220414.2>
      imagePullPolicy: IfNotPresent
      name: f11qb62y-n1-0-dn0-0-n2-0
      resources:
        limits:
          cpu: "16"
          ephemeral-storage: 52Gi
          memory: 57Gi
        requests:
          cpu: "16"
          ephemeral-storage: 26Gi
          memory: 57Gi
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: FallbackToLogsOnError
      volumeMounts:
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-6h4gh
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    nodeName: gke-orchid-west1-flyte-worker-a3b955b5-pccx
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Never
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: default
    serviceAccountName: default
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoSchedule
      key: <http://k8s.freenome.net/node-role|k8s.freenome.net/node-role>
      operator: Equal
      value: flyte-worker
    - effect: NoExecute
      key: <http://node.kubernetes.io/not-ready|node.kubernetes.io/not-ready>
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: <http://node.kubernetes.io/unreachable|node.kubernetes.io/unreachable>
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - name: kube-api-access-6h4gh
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2022-04-19T07:30:49Z"
      reason: PodCompleted
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2022-04-19T08:24:46Z"
      reason: PodCompleted
      status: "False"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2022-04-19T08:24:46Z"
      reason: PodCompleted
      status: "False"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2022-04-19T07:30:49Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: <containerd://7a10fb36a46f0c245ee20736b470f2dd271755c0d50eb5b9c0e38e0a801b6d2>0
      image: <http://gcr.io/freenome-build/ap/sunflower:20220414.2|gcr.io/freenome-build/ap/sunflower:20220414.2>
      imageID: <http://gcr.io/freenome-build/ap/sunflower@sha256:c1029318c0902a1a7c69cc9252c11c0c745d6cc5433857bc06d83b98885742a5|gcr.io/freenome-build/ap/sunflower@sha256:c1029318c0902a1a7c69cc9252c11c0c745d6cc5433857bc06d83b98885742a5>
      lastState: {}
      name: f11qb62y-n1-0-dn0-0-n2-0
      ready: false
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: <containerd://7a10fb36a46f0c245ee20736b470f2dd271755c0d50eb5b9c0e38e0a801b6d2>0
          exitCode: 0
          finishedAt: "2022-04-19T08:24:45Z"
          reason: Completed
          startedAt: "2022-04-19T07:30:49Z"
    hostIP: 172.31.0.27
    phase: Succeeded
    podIP: 172.20.53.23
    podIPs:
    - ip: 172.20.53.23
    qosClass: Guaranteed
    startTime: "2022-04-19T07:30:49Z"
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

I exec’d into the flytepropeller pod and here are the contents of all our config yamls:

flytepropeller_config.yaml

To join the video meeting, click this link: https://meet.google.com/yvu-kffs-gqh Otherwise, to join by phone, dial +1 720-598-2817 and enter this PIN: 193 537 382# To view more phone numbers, click this link: https://tel.meet/yvu-kffs-gqh?hs=5

So then intuitively propeller just is not processing the nodes, and in turn removing the finalizers. It's most likely not a bug, just a bottleneck. It would probably benefit to look at some prometheus metrics on the etcd query times and workflow round latencies in flytepropeller to see if it's k8s or Flyte and what we can do to reduce it.

@abundant-laptop-47033 is there a specific reason you are using launchplans instead of just subworkflows (beyond stress testing)? I ask because I don't believe the max parallelism configuration from the parent workflow is adopted by launchplans. Therefore, each workflow may execute 3 launchplans in parallel, each of which executes 3 tasks. When you extrapolate that by the 800 workflows it's something like 7200 pods simultaneously. With the resource requests on the sample pod (16 cpu / 51Gi memory) that adds up quickly.

in terms of the prometheus metrics, I found the prometheus dashboard for flytepropeller, here are some screenshots of round latency and etcd in the last 24 hours:

I think maybe I was just overloading the Kubernetes API. I reduced workers drastically (to 10 instead of 1000) and also reduced the kube-client-config qps, and suddenly the pods stuck in Terminating started getting cleaned up. Do you have tips on how to balance FlytePropeller performance with Kubernetes constraints?

I ended up landing at 75 workers and 100qps