acoustic-carpenter-78188
10/15/2023, 9:04 PMflyte:// file system and improve remote file handling
Pull request opened by wild-endeavor
TL;DR
Examples of improvements
• pyflyte run used to use a special way of handling certain objects, now that is centralized to use the typeEngine
• pyflyte run and gate node used different string parsing systems, now that is centralized
• pyflyte run can now use multiple files in a directory locally to create a new remote FlyteDirectory representation
• pyflyte run code has been simplified
• workflow defaults that point to local files/folders are auto-uploaded and includes limited support for dataframes. Also data is maintained using hashes so enables versioning
New additions
• pyflyte info added
• pyflyte fetch added
• This introduces new flyte-uri concept that is available in the UI and you can retrieve inputs and outputs from any execution (workflow or even a single node) instantaneously as it deeplinks into it.
centralizes fsspec as the way to interact with data
Planned work
• Artifacts service that is unblocked and simplified with this
• improved launchform experience from flyte uris
and many more data interaction improvements
Details
What it does?
• adds a new fsspec file system where flyte:// is the protocol. This is not a real filesystem though, so it's only used in pyflyte run and pyflyte register.
• cleans up code around data persistence.
The reason the file system is useful:
• Defaults for workflows that are of offloaded types can get uploaded properly during pyflyte run/register.
• pyflyte run currently duplicates a bunch of work that's supposed to be handled by the type engine because it needs to handle remote upload. This can be cleaned up in the future if the data proxy service can be accessible at a lower level.
The drawback of this approach:
• The data proxy layer is not a real filesystem. Currently you can't know beforehand where the thing you upload will land. That is, if you think about writing files as a cp src flyte://dest command, the destination is ignored. For this reason, rpath in the implementation is always ignored and a placeholder is used instead. Instead, the remote_fs implementation returns the s3 native url as part of the put call.
• If your default input is a file/folder/pickle, it works, but if it's a dataframe, it works, but sadly, you may end up with duplicate registration problems, the reasons are as follows,
This is basically an fsspec filesystem that writes to a different location than what is specified. However, it's not always possible to capture the output where the file was really written to. More concretely, if you do:
df.to_parquet("s3://a/b/c.parquet")
pandas will use fsspec to fetch the file system, call open on it, and then write to the open buffer. After writing, users expect that s3 location to hold the contents of the dataframe. With the flyte remote fs, if you called df.to_parquet("flyte://anything"), the dataframe contents do not end up there. Instead they end up in some s3:// location (assuming s3 is the blob store).
Because of this, calls to put_data were updated to read back the remote_path, now returned by put_data. (this is why dataframes don't work as default input. we can of course add another transformer that writes to a memory buffer or file first of course only for pyflyte run/register.)
To promote the behavior, a new function was added to data_persistence put_raw_data which writes always to the raw output data prefix. See the function and corresponding test. This PR does not update all the calls yet to the new interface.
The flyte fs also does not support open. The reason is simply that the data proxy call requires that we know the MD5 hash. This doesn't really make sense in the context of an open call where the data might not even be known yet. If this wasn't the case, we could've written the pandas transformer to fs.open() and then write directly to the buffer.
Also,
• The old get_random_remote_path/directory was removed from data persistence and rewritten to be based on an input fs.
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☑︎ Code documentation added
☑︎ Any pending items have an associated Issue
Complete description
Smaller changes
• Add length to hash_file
• Added a limited-use (only Remote uses it for now) local fsspec implementation which overrides the / field for proper Windows use.
Other thoughts
We also briefly thought about replacing the fsspec s3 filesystem with something that will do the signed urls but this felt more complicated and felt like it interfered too much with fsspec. Also would be dependent on what blob store admin was configured to use.
Needs: flyteorg/flyteidl#416 and flyteorg/flyteadmin#577
flyteorg/flytestdlib#160 and flyteorg/flyteadmin#587
Tracking Issue
NA
flyteorg/flytekit
Codecov: 48.56% (-6.52%) compared to 1b5692a
Codecov: 21.24% of diff hit (target 55.07%)
✅ 28 other checks have passed
28/30 successful checksacoustic-carpenter-78188
10/15/2023, 9:04 PMflyte:// file system and improve remote file handling
Pull request ready for review by kumare3
flyteorg/flytekitacoustic-carpenter-78188
10/23/2023, 8:36 PMflyte:// file system and improve remote file handling
Pull request merged by kumare3
flyteorg/flytekit