#600 Add k8s events to task phase updates Pull request opened by andrewwdye TL;DR Add support for watching and sending batched kubernetes object events in

TaskExecutionEvent

Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description This change adds support for watching and sending batched kubernetes object events in

TaskExecutionEvent

. It • Adds an EventWatcher to keep a local cache of kubernetes object event adds/deletes. To be mindful of memory, just the event note and timestamp are stored. To avoid bloating the task closure for spammy events, only stores the creation timestamp of the event (first occurrence). Others are ignored. This tradeoff means that information about current state may not be available in the UI (i.e., crash looping) • Refactors clientset plumbing to use a separate clientset for the event informer. The flyte clientset abstraction does not support filtering to specific object types. • Sends object events to admin via the newly added batched

TaskExecutionEvent.reasons

field • This feature is default disabled (turned on via

K8sPluginConfig.SendObjectEvents

) Testing Ran single binary locally and got the following console output

9/29/2023 4:50:59 PM UTC task submitted to K8s

9/29/2023 4:50:59 PM UTC Unschedulable:0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

9/29/2023 4:50:59 PM UTC 0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

9/29/2023 4:52:11 PM UTC [ContainersNotReady|ContainerCreating]: containers with unready status: [f4e1edfa9f2a045bb9cd-n0-0]|

9/29/2023 4:52:11 PM UTC Successfully assigned flytesnacks-development/f4e1edfa9f2a045bb9cd-n0-0 to 5b0f2dc6442c

9/29/2023 4:52:12 PM UTC Container image "<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.8.1|cr.flyte.org/flyteorg/flytekit:py3.10-1.8.1>" already present on machine

9/29/2023 4:52:12 PM UTC Created container f4e1edfa9f2a045bb9cd-n0-0

9/29/2023 4:52:12 PM UTC Started container f4e1edfa9f2a045bb9cd-n0-0

Looking at the execution closure

❯ flytectl get execution -p flytesnacks -d development f4e1edfa9f2a045bb9cd --details -o yaml
- node_exec:
    closure:
      createdAt: "2023-09-29T16:50:59.879001Z"
      outputUri: <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f4e1edfa9f2a045bb9cd/start-node/data/0/outputs.pb>
      phase: SUCCEEDED
      updatedAt: "2023-09-29T16:50:59.879041Z"
    id:
      executionId:
        domain: development
        name: f4e1edfa9f2a045bb9cd
        project: flytesnacks
      nodeId: start-node
    inputUri: <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f4e1edfa9f2a045bb9cd/start-node/data/inputs.pb>
    metadata:
      specNodeId: start-node
- node_exec:
    closure:
      createdAt: "2023-09-29T16:50:59.886835Z"
      phase: RUNNING
      startedAt: "2023-09-29T16:50:59.909353Z"
      updatedAt: "2023-09-29T16:50:59.909698Z"
    id:
      executionId:
        domain: development
        name: f4e1edfa9f2a045bb9cd
        project: flytesnacks
      nodeId: n0
    inputUri: <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f4e1edfa9f2a045bb9cd/n0/data/inputs.pb>
    metadata:
      specNodeId: n0
  task_execs:
  - closure:
      createdAt: "2023-09-29T16:50:59.903548Z"
      eventVersion: 1
      logs:
      - messageFormat: JSON
        name: Kubernetes Logs (User)
        uri: <http://localhost:30080/kubernetes-dashboard/#/log/flytesnacks-development/f4e1edfa9f2a045bb9cd-n0-0/pod?namespace=flytesnacks-development>
      metadata:
        generatedName: f4e1edfa9f2a045bb9cd-n0-0
        pluginIdentifier: container
      phase: RUNNING
      reason: Started container f4e1edfa9f2a045bb9cd-n0-0
      reasons:
      - message: task submitted to K8s
        occurredAt: "2023-09-29T16:50:59.903548Z"
      - message: 'Unschedulable:0/1 nodes are available: 1 Insufficient cpu. preemption:
          0/1 nodes are available: 1 No preemption victims found for incoming pod.'
        occurredAt: "2023-09-29T16:50:59Z"
      - message: '0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes
          are available: 1 No preemption victims found for incoming pod.'
        occurredAt: "2023-09-29T16:50:59Z"
      - message: '[ContainersNotReady|ContainerCreating]: containers with unready
          status: [f4e1edfa9f2a045bb9cd-n0-0]|'
        occurredAt: "2023-09-29T16:52:11Z"
      - message: Successfully assigned flytesnacks-development/f4e1edfa9f2a045bb9cd-n0-0
          to 5b0f2dc6442c
        occurredAt: "2023-09-29T16:52:11Z"
      - message: Container image "<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.8.1|cr.flyte.org/flyteorg/flytekit:py3.10-1.8.1>" already
          present on machine
        occurredAt: "2023-09-29T16:52:12Z"
      - message: Created container f4e1edfa9f2a045bb9cd-n0-0
        occurredAt: "2023-09-29T16:52:12Z"
      - message: Started container f4e1edfa9f2a045bb9cd-n0-0
        occurredAt: "2023-09-29T16:52:12Z"
      startedAt: "2023-09-29T16:52:12Z"
      taskType: python-task
      updatedAt: "2023-09-29T16:52:12.770241Z"

Tracking Issue flyteorg/flyte#3825 Follow-up issue N/A flyteorg/flytepropeller GitHub Actions: Goreleaser GitHub Actions: Build & Push Flytepropeller Image GitHub Actions: Bump Version ✅ 11 other checks have passed 11/14 successful checks