<#1674 Flyte `flyte://` file system and improve re...
# flyte-github
a
#1674 Flyte
flyte://
file system and improve remote file handling
Pull request opened by wild-endeavor TL;DR Examples of improvements • pyflyte run used to use a special way of handling certain objects, now that is centralized to use the typeEngine • pyflyte run and gate node used different string parsing systems, now that is centralized • pyflyte run can now use multiple files in a directory locally to create a new remote FlyteDirectory representation • pyflyte run code has been simplified • workflow defaults that point to local files/folders are auto-uploaded and includes limited support for dataframes. Also data is maintained using hashes so enables versioning New additions • pyflyte info added • pyflyte fetch added • This introduces new flyte-uri concept that is available in the UI and you can retrieve inputs and outputs from any execution (workflow or even a single node) instantaneously as it deeplinks into it. centralizes fsspec as the way to interact with data Planned work • Artifacts service that is unblocked and simplified with this • improved launchform experience from flyte uris and many more data interaction improvements Details What it does? • adds a new fsspec file system where
flyte://
is the protocol. This is not a real filesystem though, so it's only used in
pyflyte run
and
pyflyte register
. • cleans up code around data persistence. The reason the file system is useful: • Defaults for workflows that are of offloaded types can get uploaded properly during pyflyte run/register. •
pyflyte run
currently duplicates a bunch of work that's supposed to be handled by the type engine because it needs to handle remote upload. This can be cleaned up in the future if the data proxy service can be accessible at a lower level. The drawback of this approach: • The data proxy layer is not a real filesystem. Currently you can't know beforehand where the thing you upload will land. That is, if you think about writing files as a
cp src flyte://dest
command, the destination is ignored. For this reason,
rpath
in the implementation is always ignored and a placeholder is used instead. Instead, the
remote_fs
implementation returns the s3 native url as part of the
put
call. • If your default input is a file/folder/pickle, it works, but if it's a dataframe, it works, but sadly, you may end up with duplicate registration problems, the reasons are as follows, This is basically an fsspec filesystem that writes to a different location than what is specified. However, it's not always possible to capture the output where the file was really written to. More concretely, if you do:
Copy code
df.to_parquet("s3://a/b/c.parquet")
pandas will use fsspec to fetch the file system, call
open
on it, and then write to the open buffer. After writing, users expect that s3 location to hold the contents of the dataframe. With the flyte remote fs, if you called
df.to_parquet("flyte://anything")
, the dataframe contents do not end up there. Instead they end up in some
s3://
location (assuming s3 is the blob store). Because of this, calls to
put_data
were updated to read back the
remote_path
, now returned by
put_data
. (this is why dataframes don't work as default input. we can of course add another transformer that writes to a memory buffer or file first of course only for pyflyte run/register.) To promote the behavior, a new function was added to data_persistence
put_raw_data
which writes always to the raw output data prefix. See the function and corresponding test. This PR does not update all the calls yet to the new interface. The flyte fs also does not support
open
. The reason is simply that the data proxy call requires that we know the MD5 hash. This doesn't really make sense in the context of an open call where the data might not even be known yet. If this wasn't the case, we could've written the pandas transformer to fs.open() and then write directly to the buffer. Also, • The old
get_random_remote_path/directory
was removed from data persistence and rewritten to be based on an input fs. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Complete description Smaller changes • Add length to
hash_file
• Added a limited-use (only Remote uses it for now) local fsspec implementation which overrides the
/
field for proper Windows use. Other thoughts We also briefly thought about replacing the fsspec s3 filesystem with something that will do the signed urls but this felt more complicated and felt like it interfered too much with fsspec. Also would be dependent on what blob store admin was configured to use. Needs: flyteorg/flyteidl#416 and flyteorg/flyteadmin#577 flyteorg/flytestdlib#160 and flyteorg/flyteadmin#587 Tracking Issue NA flyteorg/flytekit Codecov: 48.56% (-6.52%) compared to 1b5692a Codecov: 21.24% of diff hit (target 55.07%) ✅ 28 other checks have passed 28/30 successful checks