Is there any way to tune the parallelism for local...
# flyte-supporte
elegant-australia-91422
10/20/2023, 2:24 PMIs there any way to tune the parallelism for locally-executed workflows/run various tasks in parallel as opposed to serially?
elegant-australia-91422
10/20/2023, 11:10 PM@freezing-airport-6809 @high-accountant-32689 would you be game for a joblib executor for parallel local runs? This piece is something we’d love to contribute (dask integration wound up deprioritized on our side)
f
freezing-airport-6809
10/20/2023, 11:14 PM
if you are open to contributing absolutely
freezing-airport-6809
10/20/2023, 11:14 PM
👍
e
elegant-australia-91422
10/20/2023, 11:15 PMDefinitely. I’m not sure about implementation details but we run JupyterHub servers w/ 100+ GB of RAM and we’d love to parallelize execution for our researchers (especially back-testing workflows which perform a lot of parallel training steps)
elegant-australia-91422
10/20/2023, 11:16 PMHappy to chat in the next week or two if the team has existing thoughts
f
freezing-airport-6809
10/20/2023, 11:20 PM
btw have you seen this - https://github.com/flyteorg/flyte/issues/4205
👀 1
e
elegant-australia-91422
10/20/2023, 11:22 PMWe’d prefer to not execute remotely given the overhead most likely. Something like a shared cache for both remote and local is very interesting though (a lot like Bazel’s remote build caching)
elegant-australia-91422
10/20/2023, 11:23 PMThink this is a bit different from the agents use case tho
f
freezing-airport-6809
10/20/2023, 11:40 PM
ya this is not agent
freezing-airport-6809
10/20/2023, 11:40 PM
this is to send things remotely
freezing-airport-6809
10/20/2023, 11:41 PM
but we are all rowing in the same direction
freezing-airport-6809
10/20/2023, 11:41 PM
powerful local, then offloading and then remote
t
thankful-minister-83577
10/21/2023, 1:45 AM
@elegant-australia-91422 i’ve been meaning to investigate this. i think we can add a parallel executor in flytekit so that all the tasks at least run async if not parallel. might be one of the things i work on after the current project.
🔝 1
f
freezing-airport-6809
10/21/2023, 1:46 AM
@elegant-australia-91422 we would love help in first making everything async
e
elegant-australia-91422
10/21/2023, 5:02 AMAwesome. I’m mostly OOO next week but @thankful-minister-83577 @freezing-airport-6809 game to chat week after next? Let’s align on what y’all are thinking and happy to land a few contributions to make this possible
elegant-australia-91422
10/21/2023, 5:03 AMBig picture, we want to make cheap experiments accessible to the research team to prioritize what to run on the full prod datasets in the cluster
elegant-australia-91422
10/21/2023, 5:03 AMSo both sides of the equation are important
10 Views