Folks we need help, we are working on some core ch...
# announcementsk
Ketan (kumare3)
08/04/2022, 3:30 PM
Folks we need help, we are working on some core changes that will potentially improve performance of the core engine upto 100x, so a pure cached workflow will complete in less than a second, all transitions will be faster.
Moreover in collaboration with Spotify we are adding a feature that makes it possible to have absolutely gigantic workflows. We would love volunteers who can help us test and benchmark it? Highly appreciated
🔥 5
💯 4
Ketan (kumare3)
08/04/2022, 3:31 PM
Cc @Babis Kiosidis @Sonja Ericsson @jeev @Pradithya Aria Pura @Arief Rahmansyah @Andrew Achkar @Rahul Mehta @Stephen
Ketan (kumare3)
08/04/2022, 3:31 PM
And all others i just put a few names here
r
Rahul Mehta
08/04/2022, 3:31 PMDefinitely!
❤️ 1
k
Ketan (kumare3)
08/04/2022, 3:31 PM
Cc @Prada Souvanlasy
s
Stephen
08/04/2022, 3:31 PMFor sure!
❤️ 1
k
Ketan (kumare3)
08/04/2022, 3:32 PM
@Rahul Mehta how should we go about sharing these updates, if we cut a beta release would folks be ready to try
r
Rahul Mehta
08/04/2022, 3:32 PMSure; would this involve a beta release of the backend services along with flytekit? Happy to upgrade our flytekit requirement to a beta release, and for the backend services would it be possible to tag a beta release of the helm chart?
k
Ketan (kumare3)
08/04/2022, 3:33 PM
No changes to flytekit, pure backend only change
👍 1
Ketan (kumare3)
08/04/2022, 3:34 PM
It will be atleast a week or so more
Ketan (kumare3)
08/04/2022, 3:34 PM
But we will create a channel of beta adopters to help improve release process - cc @Dan Rammer (hamersaw) / @Yee
👍 1
Ketan (kumare3)
08/04/2022, 3:34 PM
Also cc @Maarten de Jong
Ketan (kumare3)
08/04/2022, 3:34 PM
@Nick Müller (MorpheusXAUT)
Ketan (kumare3)
08/04/2022, 3:37 PM
Cc @Dylan Wilder also because you raised some concerns recently- you will really like the results I think
Ketan (kumare3)
08/04/2022, 3:37 PM
It goes pretty fast
n
Nick Müller (MorpheusXAUT)
08/04/2022, 3:43 PMSounds awesome, people will sure be glad about that 🙂 is that something we could temporarily replace our current flyte clusters with for benchmarking, or would you recommend we create a fresh, separate setup for that?
Nick Müller (MorpheusXAUT)
08/04/2022, 3:45 PMin any case, happy to help out with testing and some benchmarking
k
Ketan (kumare3)
08/04/2022, 3:58 PM
You could
Replace temporarily
👍 1
n
Nick Müller (MorpheusXAUT)
08/04/2022, 3:59 PMPerfekt
d
Dylan Wilder
08/04/2022, 4:05 PMhey we would definitely like to see this, but what's involved in benchmarking? we don't have a sandbox cluster personally, but if spotify flyte admins have one we definitely have some test workflows 😉
k
Ketan (kumare3)
08/04/2022, 4:23 PM
you wont need to benchmark we will work with the infra team
Ketan (kumare3)
08/04/2022, 4:23 PM
yes they will
Ketan (kumare3)
08/04/2022, 4:23 PM
for most cached cases we want the entire thing to be really fast
b
Babis Kiosidis
08/04/2022, 4:36 PMYep we are working together on some parts of this
j
jeev
08/04/2022, 11:16 PM
we'd love to partner on this @Ketan (kumare3)
❤️ 1
k
Ketan (kumare3)
08/05/2022, 4:30 AM
I will create a channel - stay tuned
n
Nick Müller (MorpheusXAUT)
08/19/2022, 8:00 AMhas there been any news about this lately @Ketan (kumare3)? 🙂 as we're preparing for a larger scale run, this would be interesting to test and see how it affects our scheduling
k
Ketan (kumare3)
08/19/2022, 2:46 PM
Ohh ya, but we have merged some changes couple still are wip
Ketan (kumare3)
08/19/2022, 2:46 PM
But Nick let me send a few suggestions and you can help try it out
157 Views