#6970 [V2][Feat] Add events service Pull request opened by machichima Tracking issue Why are the changes needed? Previous we call internal run service to record action events (persist to DB) from action client (k8s watcher). However, we cannot get the full info we want from the CR (e.g.

LogInfo

,

ErrorInfo

, ...). In this PR, we add a new

EventService

running independently, receiving requests from executor and forward to internal run service through background workers. Alternatives We've considered following alternatives: 1. Store map of

ActionEvents

under

TaskAction

CR status We can store a map of action events with ID under the

TaskAction

CR status, and add a

persisted_id

slice to record the persisted action events. Executor will put new action event to the CR, and action service will take the action event out -> check if it's persisted by checking

persisted_id

, and call request internal run service. This is using

TaskAction

CR as a queue, which we need to: 1. take the action event info out 2. determine which action event is persisted 3. remove the persisted action event This method has following disadvantages: 1. can be more complex, as we need to handle the queueing logic on our own 2. may cause performance issue in large scale 3. Using CR object as queue is not a good practice 2. Store

ActionEvent

in a new CR We can add a new

ActionEvent

CR to store the action events. Executor will create a new

ActionEvent

CR when there's status update, and action service can watch this CR. Whenever there's new

ActionEvent

CR create, it will parse info from the CR and send request to internal run service for persisting action event. This method has following disadvantages: 1. If run service is down, we may create overwhelming CRs, causing pressure to etcd What changes were proposed in this pull request? • Add new

EventsService

proto and implementations • Start a background goroutine

EventsWorker

to queue and send requests to internal run service (through unary) • Clean-up

executor/cmd/main.go

and remove unused code TODO next: Use streaming endpoint / implement batch request from

EventsService

to internal run service How was this patch tested? 1. Start Flyte locally following

manager/DEVELOPMENT.md

2. Use this config in flyte-sdk admin: endpoint: dns///localhost8090 insecure: True image: builder: local task: domain: development project: testproject org: testorg 1. Run an example (e.g.

python examples/basics/hello_v2.py

) 2. Look into the SQLite DB (

flyte.db

), check if the data exists image DB: image Labels Please add one or more of the following labels to categorize your PR: • added: For new features. • changed: For changes in existing functionality. • deprecated: For soon-to-be-removed features. • removed: For features being removed. • fixed: For any bug fixed. • security: In case of vulnerabilities This is important to improve the readability of release notes. Setup process Screenshots Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. Related PRs Docs link •

main

• #6583 • #6970 👈 flyteorg/flyte