#1526 Data subsystem Pull request opened by wild-endeavor TL;DR This replaces the flytekit custom persistence layer and routes all calls to fsspec directly. Currently still relying on

s3fs/gcsfs

to provide the fsspec implementations. Important This PR removes the

DataPersistence

and

DataPersistencePlugins

constructs completely. If you were importing these, these are now going away. However most people should not have been using these. The

flytekitplugins-data-fsspec

plugin has also been emptied out, though it will continue to be published for the time being. The logic that was in there originally has been moved into flytekit code in this PR, including the Structured Dataset handlers. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Notes for users and contributors If you're importing

LOCAL

or

S3

from

flytekit/types/structured/structured_dataset.py

these have been removed. If you're registering with

None

as the protocol for a structured dataset handler, what happens now is that the handler is registered with the

fsspec

protocol. For users that are not familiar with

StructuredDataset

encoders and decoders, which one gets used is based on three things: the Python type of the dataclass, the storage protocol (local vs s3 vs gcs etc.) and the file format (parquet, csv, etc.). This was done as a separate commit #1543. The waterfall is now: 1. Exact match on all three attributes 2. Protocol match but a generic format (

""

) handler, if registered 3. Protocol match and a match on the default format for that dataframe type, if default format is set 4. An

fsspec

handler with a format match (will match generic

""

format too) 5. An

fsspec

handler with a generic format, if registered 6. An

fsspec

handler with the default format for that dataframe type, if default format is set 7. An

fsspec

handler if that protocol only has one entry and the requested format was generic. 8. Protocol specific handler if that protocol only has one entry. Others • Added a

Dockerfile.dev

for slightly easier debugging. (Longer term we hope to come up with a better dev experience.) • Some attempt was made to add the ability to include a custom header containing a hash to the fsspec HTTP filesystem. This is needed for

pyflyte run

register

and anything else that uses the

DataProxy

service to post upload to a signed URL. Unfortunately no solution was found, so the upload path for these two commands just uses the old way of calling the requests library. Tracking Issue flyteorg/flyte#3197 flyteorg/flytekit ✅ All checks have passed 30/30 successful checks