Is there any way to tune the parallelism for locally-executed workflows/run various tasks in parallel as opposed to serially?

if you are open to contributing absolutely

Definitely. I’m not sure about implementation details but we run JupyterHub servers w/ 100+ GB of RAM and we’d love to parallelize execution for our researchers (especially back-testing workflows which perform a lot of parallel training steps)

btw have you seen this - https://github.com/flyteorg/flyte/issues/4205

We’d prefer to not execute remotely given the overhead most likely. Something like a shared cache for both remote and local is very interesting though (a lot like Bazel’s remote build caching)

ya this is not agent

@Rahul Mehta i’ve been meaning to investigate this. i think we can add a parallel executor in flytekit so that all the tasks at least run async if not parallel. might be one of the things i work on after the current project.

@Rahul Mehta we would love help in first making everything async

Awesome. I’m mostly OOO next week but @Yee @Ketan (kumare3) game to chat week after next? Let’s align on what y’all are thinking and happy to land a few contributions to make this possible