Bernhard Stadlbauer
05/18/2022, 8:54 AMPythonTask.pre_execute()
, so user_params: ExecutionParameters
would also be available if that helps.Yee
Ketan (kumare3)
Bernhard Stadlbauer
05/19/2022, 5:47 AMflytekit
dask
plugin at the moment (very similar to this Flyte issue). I thought it might be nice if the dask
cluster spins up its scheduler and worker nodes with the same image as the one that the Flyte task is running on (in the remote execution case).
As a side note, if we things are working, we would also like to upstream our changes and contribute the plugin to the flytekit
repository. I've currently based my work on the new (and experimental) dask operator.Ketan (kumare3)
Marc Paquette
05/19/2022, 3:04 PMBernhard Stadlbauer
05/19/2022, 5:46 PMflytekit
plugin, which is only a few lines of code) 👍
@Marc Paquette Sorry for that, this should be the link: https://github.com/flyteorg/flyte/issues/427Marc Paquette
05/19/2022, 5:46 PMKetan (kumare3)