https://flyte.org logo
k

Ketan (kumare3)

08/04/2022, 3:30 PM
Folks we need help, we are working on some core changes that will potentially improve performance of the core engine upto 100x, so a pure cached workflow will complete in less than a second, all transitions will be faster. Moreover in collaboration with Spotify we are adding a feature that makes it possible to have absolutely gigantic workflows. We would love volunteers who can help us test and benchmark it? Highly appreciated
🔥 5
💯 4
Cc @Babis Kiosidis @Sonja Ericsson @jeev @Pradithya Aria Pura @Arief Rahmansyah @Andrew Achkar @Rahul Mehta @Stephen
And all others i just put a few names here
r

Rahul Mehta

08/04/2022, 3:31 PM
Definitely!
❤️ 1
k

Ketan (kumare3)

08/04/2022, 3:31 PM
Cc @Prada Souvanlasy
s

Stephen

08/04/2022, 3:31 PM
For sure!
❤️ 1
k

Ketan (kumare3)

08/04/2022, 3:32 PM
@Rahul Mehta how should we go about sharing these updates, if we cut a beta release would folks be ready to try
r

Rahul Mehta

08/04/2022, 3:32 PM
Sure; would this involve a beta release of the backend services along with flytekit? Happy to upgrade our flytekit requirement to a beta release, and for the backend services would it be possible to tag a beta release of the helm chart?
k

Ketan (kumare3)

08/04/2022, 3:33 PM
No changes to flytekit, pure backend only change
👍 1
It will be atleast a week or so more
But we will create a channel of beta adopters to help improve release process - cc @Dan Rammer (hamersaw) / @Yee
👍 1
Also cc @Maarten de Jong
@Nick Müller (MorpheusXAUT)
Cc @Dylan Wilder also because you raised some concerns recently- you will really like the results I think
It goes pretty fast
n

Nick Müller (MorpheusXAUT)

08/04/2022, 3:43 PM
Sounds awesome, people will sure be glad about that 🙂 is that something we could temporarily replace our current flyte clusters with for benchmarking, or would you recommend we create a fresh, separate setup for that?
in any case, happy to help out with testing and some benchmarking
k

Ketan (kumare3)

08/04/2022, 3:58 PM
You could Replace temporarily
👍 1
n

Nick Müller (MorpheusXAUT)

08/04/2022, 3:59 PM
Perfekt
d

Dylan Wilder

08/04/2022, 4:05 PM
hey we would definitely like to see this, but what's involved in benchmarking? we don't have a sandbox cluster personally, but if spotify flyte admins have one we definitely have some test workflows 😉
k

Ketan (kumare3)

08/04/2022, 4:23 PM
you wont need to benchmark we will work with the infra team
yes they will
for most cached cases we want the entire thing to be really fast
b

Babis Kiosidis

08/04/2022, 4:36 PM
Yep we are working together on some parts of this
j

jeev

08/04/2022, 11:16 PM
we'd love to partner on this @Ketan (kumare3)
❤️ 1
k

Ketan (kumare3)

08/05/2022, 4:30 AM
I will create a channel - stay tuned
n

Nick Müller (MorpheusXAUT)

08/19/2022, 8:00 AM
has there been any news about this lately @Ketan (kumare3)? 🙂 as we're preparing for a larger scale run, this would be interesting to test and see how it affects our scheduling
k

Ketan (kumare3)

08/19/2022, 2:46 PM
Ohh ya, but we have merged some changes couple still are wip
But Nick let me send a few suggestions and you can help try it out