straight-laptop-71325
05/20/2022, 6:26 PMacceptable-policeman-57188
broad-monitor-993
05/20/2022, 7:57 PMList[int]
or List[float]
and reconstituting your arrays/tensors at the beginning of the next task
3. using a np.ndarray
or torch.Tensor
annotation purely for human-readability. Under the hood this will pickle your array/tensor and unpickle it on the other side.
(3) is convenient, but you run the risk of deserialization issues if you happen to use different versions of python/numpy/pytorch/tensorflow across your tasks that are not cross-compatible. (2) is really for smaller data use cases since these are stored as FlyteIDL literals. (1) is nice because flyte understands this and stores dataframes as parquet files, which is a more efficient/reliable storage format than pickle.straight-laptop-71325
05/20/2022, 8:11 PMfreezing-airport-6809
tall-lock-23197