Folks we need help, we are working on some core changes that will potentially improve performance of the core engine upto 100x, so a pure cached workflow will complete in less than a second, all transitions will be faster. Moreover in collaboration with Spotify we are adding a feature that makes it possible to have absolutely gigantic workflows. We would love volunteers who can help us test and benchmark it? Highly appreciated

Definitely!

Cc @limited-sundown-57683

For sure!

@elegant-australia-91422 how should we go about sharing these updates, if we cut a beta release would folks be ready to try

Sure; would this involve a beta release of the backend services along with flytekit? Happy to upgrade our flytekit requirement to a beta release, and for the backend services would it be possible to tag a beta release of the helm chart?

No changes to flytekit, pure backend only change

Sounds awesome, people will sure be glad about that 🙂 is that something we could temporarily replace our current flyte clusters with for benchmarking, or would you recommend we create a fresh, separate setup for that?

You could Replace temporarily

Perfekt

hey we would definitely like to see this, but what's involved in benchmarking? we don't have a sandbox cluster personally, but if spotify flyte admins have one we definitely have some test workflows 😉

you wont need to benchmark we will work with the infra team

Yep we are working together on some parts of this

we'd love to partner on this @freezing-airport-6809

I will create a channel - stay tuned

has there been any news about this lately @freezing-airport-6809? 🙂 as we're preparing for a larger scale run, this would be interesting to test and see how it affects our scheduling

Ohh ya, but we have merged some changes couple still are wip