so it should be possible to pickle and run tasks d...
# torch-elastick
Ketan (kumare3)
04/04/2023, 5:06 AM
so it should be possible to pickle and run tasks directly from notebook onto a flyte-server
g
Greg Gydush
04/04/2023, 5:07 AMthat's epic
Greg Gydush
04/04/2023, 5:07 AMI'm imagining map tasks from within notebook would be incredible
k
Ketan (kumare3)
04/04/2023, 5:07 AM
yup
Ketan (kumare3)
04/04/2023, 5:07 AM
exactly
Ketan (kumare3)
04/04/2023, 5:07 AM
also distributed training
Ketan (kumare3)
04/04/2023, 5:07 AM
and others
Ketan (kumare3)
04/04/2023, 5:08 AM
so i think we should just rewrite this resolverrr https://github.com/flyteorg/flytekit/blob/master/flytekit/extras/cloud_pickle_resolver.py
Ketan (kumare3)
04/04/2023, 5:08 AM
or use this resolver to use in fast-execute
Ketan (kumare3)
04/04/2023, 5:08 AM
and update pyflyte-fast-execute to use cloudpickle as an option
Ketan (kumare3)
04/04/2023, 5:08 AM
and then it should just work IMO?
Ketan (kumare3)
04/04/2023, 5:09 AM
Also @Greg Gydush have you seen the new map-tasks
Ketan (kumare3)
04/04/2023, 5:09 AM
support for multiple-lists etc and partials?
g
Greg Gydush
04/04/2023, 5:09 AMI haven't yet
k
Ketan (kumare3)
04/04/2023, 5:09 AM
this is in beta, some small bugs being fixed, but keep an eye
Ketan (kumare3)
04/04/2023, 5:10 AM
154 Views