#485 Move caching to node executor for fast cache hits
Pull request opened by
hamersaw
TL;DR
Currently cache lookups require 3 node-level phase changes, which equates to 3 streak rounds of flytepropeller execution. The overhead of updating the k8s FlyteWorkflow CRD after each of these rounds may make it seem like cache lookups are slow. This PR updates cache lookups to happen in the node executor, which allows us to transition nodes directly from NotYetStarted to Success and immediately process downstream nodes in the same FlytePropeller round making cache lookup overhead minimal.
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
In addition to the performance improvements this PR refactors the code to support future implementation of caching other Flyte node types (ie. subworkflow / launchplan and array nodes when they are implemented).
Tracking Issue
fixes
flyteorg/flyte#2886
Follow-up issue
NA
flyteorg/flytepropeller
GitHub Actions: Build & Push Flytepropeller Image
GitHub Actions: Goreleaser
GitHub Actions: Bump Version
✅ 11 other checks have passed
11/14 successful checks