Is there any way to tune the parallelism for local...
# ask-the-communityr
Rahul Mehta
10/20/2023, 2:24 PMIs there any way to tune the parallelism for locally-executed workflows/run various tasks in parallel as opposed to serially?
Rahul Mehta
10/20/2023, 11:10 PM@Ketan (kumare3) @Eduardo Apolinario (eapolinario) would you be game for a joblib executor for parallel local runs? This piece is something we’d love to contribute (dask integration wound up deprioritized on our side)
k
Ketan (kumare3)
10/20/2023, 11:14 PM
if you are open to contributing absolutely
Ketan (kumare3)
10/20/2023, 11:14 PM
👍
r
Rahul Mehta
10/20/2023, 11:15 PMDefinitely. I’m not sure about implementation details but we run JupyterHub servers w/ 100+ GB of RAM and we’d love to parallelize execution for our researchers (especially back-testing workflows which perform a lot of parallel training steps)
Rahul Mehta
10/20/2023, 11:16 PMHappy to chat in the next week or two if the team has existing thoughts
k
Ketan (kumare3)
10/20/2023, 11:20 PM
btw have you seen this - https://github.com/flyteorg/flyte/issues/4205
r
Rahul Mehta
10/20/2023, 11:22 PMWe’d prefer to not execute remotely given the overhead most likely. Something like a shared cache for both remote and local is very interesting though (a lot like Bazel’s remote build caching)
Rahul Mehta
10/20/2023, 11:23 PMThink this is a bit different from the agents use case tho
k
Ketan (kumare3)
10/20/2023, 11:40 PM
ya this is not agent
Ketan (kumare3)
10/20/2023, 11:40 PM
this is to send things remotely
Ketan (kumare3)
10/20/2023, 11:41 PM
but we are all rowing in the same direction
Ketan (kumare3)
10/20/2023, 11:41 PM
powerful local, then offloading and then remote
y
Yee
10/21/2023, 1:45 AM
@Rahul Mehta i’ve been meaning to investigate this. i think we can add a parallel executor in flytekit so that all the tasks at least run async if not parallel. might be one of the things i work on after the current project.
k
Ketan (kumare3)
10/21/2023, 1:46 AM
@Rahul Mehta we would love help in first making everything async
r
Rahul Mehta
10/21/2023, 5:02 AMAwesome. I’m mostly OOO next week but @Yee @Ketan (kumare3) game to chat week after next? Let’s align on what y’all are thinking and happy to land a few contributions to make this possible
Rahul Mehta
10/21/2023, 5:03 AMBig picture, we want to make cheap experiments accessible to the research team to prioritize what to run on the full prod datasets in the cluster
Rahul Mehta
10/21/2023, 5:03 AMSo both sides of the equation are important
2 Views