Can we use NFS instead of S3 in k8s cluster with F...
# announcementsa
astonishing-airport-86417
04/06/2022, 11:29 PMCan we use NFS instead of S3 in k8s cluster with Flyte?
f
freezing-airport-6809
04/07/2022, 4:08 AM
@astonishing-airport-86417 you can, but to communicate with PRopeller you have to use s3 (equivalent)
freezing-airport-6809
04/07/2022, 4:09 AM
so most data can be in NFS
a
astonishing-airport-86417
04/07/2022, 9:01 AM@freezing-airport-6809 What do you mean by “communicate with PRopeller”?
astonishing-airport-86417
04/07/2022, 9:02 AMI am looking on this scheme and do not see S3 there
f
freezing-airport-6809
04/07/2022, 4:24 PM
a
astonishing-airport-86417
04/08/2022, 11:22 AM@freezing-airport-6809 Am i right - there is impossible to use NFS directly instead of S3/Minio/GCS for cross-task data flow? (https://docs.flyte.org/en/latest/concepts/data_management.html#divedeep-data-management )
f
freezing-airport-6809
04/08/2022, 1:31 PM
It is not impossible
freezing-airport-6809
04/08/2022, 1:31 PM
Nobody has tried
freezing-airport-6809
04/08/2022, 1:31 PM
File system is an option
freezing-airport-6809
04/08/2022, 1:31 PM
Would you Be open to experiment and contribute
freezing-airport-6809
04/08/2022, 1:32 PM
If so @astonishing-airport-86417 wait till Tuesday- we are merging this https://github.com/flyteorg/flyte/pull/2265
freezing-airport-6809
04/08/2022, 1:33 PM
All backend one binary, one config and simple to test and even deploy
freezing-airport-6809
04/08/2022, 1:33 PM
And we would love to help you
freezing-airport-6809
04/08/2022, 1:34 PM
Just to explain backend supports local files https://github.com/flyteorg/flytestdlib/blob/d0a5767d30272c065a075a283fd2ba563834baca/storage/stow_store.go#L53
165 Views