Folks we need help we are working on some core changes that

Question

Folks we need help we are working on some core changes that will potentially improve performance of the core engine upto 100x so a pure cached workflow will complete in less than a second all transitions will be faster Moreover in collaboration with Spotify we are adding a feature that makes it possible to have absolutely gigantic workflows We would love volunteers who can help us test and benchmark it Highly appreciated

Ketan (kumare3) · Answer

Cc < Babis Kiosidis> < Sonja Ericsson> < jeev> < Pradithya Aria Pura> < Arief Rahmansyah> < Andrew Achkar> < Rahul Mehta> < Stephen>

Ketan (kumare3) · Answer

And all others i just put a few names here

Rahul Mehta · Answer

Definitely

Ketan (kumare3) · Answer

Cc < Prada Souvanlasy>

Stephen · Answer

For sure

Ketan (kumare3) · Answer

< Rahul Mehta> how should we go about sharing these updates if we cut a beta release would folks be ready to try

Rahul Mehta · Answer

Sure would this involve a beta release of the backend services along with flytekit Happy to upgrade our flytekit requirement to a beta release and for the backend services would it be possible to tag a beta release of the helm chart

Ketan (kumare3) · Answer

No changes to flytekit pure backend only change

Ketan (kumare3) · Answer

It will be atleast a week or so more

Ketan (kumare3) · Answer

But we will create a channel of beta adopters to help improve release process cc < Dan Rammer hamersaw > < Yee>

Ketan (kumare3) · Answer

Also cc < Maarten de Jong>

Ketan (kumare3) · Answer

< Nick Müller MorpheusXAUT >

Ketan (kumare3) · Answer

Cc < Dylan Wilder> also because you raised some concerns recently you will really like the results I think

Ketan (kumare3) · Answer

It goes pretty fast

Nick Müller (MorpheusXAUT) · Answer

Sounds awesome people will sure be glad about that slightly smiling face is that something we could temporarily replace our current flyte clusters with for benchmarking or would you recommend we create a fresh separate setup for that

Nick Müller (MorpheusXAUT) · Answer

in any case happy to help out with testing and some benchmarking

Ketan (kumare3) · Answer

You could Replace temporarily

Nick Müller (MorpheusXAUT) · Answer

Perfekt

Dylan Wilder · Answer

hey we would definitely like to see this but what s involved in benchmarking we don t have a sandbox cluster personally but if spotify flyte admins have one we definitely have some test workflows wink

Ketan (kumare3) · Answer

you wont need to benchmark we will work with the infra team

Ketan (kumare3) · Answer

yes they will

Ketan (kumare3) · Answer

for most cached cases we want the entire thing to be really fast

Babis Kiosidis · Answer

Yep we are working together on some parts of this

jeev · Answer

we d love to partner on this < Ketan kumare3 >

Ketan (kumare3) · Answer

I will create a channel stay tuned

Nick Müller (MorpheusXAUT) · Answer

has there been any news about this lately < Ketan kumare3 > slightly smiling face as we re preparing for a larger scale run this would be interesting to test and see how it affects our scheduling

Ketan (kumare3) · Answer

Ohh ya but we have merged some changes couple still are wip

Ketan (kumare3) · Answer

But Nick let me send a few suggestions and you can help try it out