Badar Ahmed05/20/2022, 6:26 PM
Niels Bantilan05/20/2022, 7:57 PM
and reconstituting your arrays/tensors at the beginning of the next task 3. using a
annotation purely for human-readability. Under the hood this will pickle your array/tensor and unpickle it on the other side. (3) is convenient, but you run the risk of deserialization issues if you happen to use different versions of python/numpy/pytorch/tensorflow across your tasks that are not cross-compatible. (2) is really for smaller data use cases since these are stored as FlyteIDL literals. (1) is nice because flyte understands this and stores dataframes as parquet files, which is a more efficient/reliable storage format than pickle.
Badar Ahmed05/20/2022, 8:11 PM