#3325 Fix: FlyteDirectory fails for multiple files with the same name Pull request opened by mys007 ## Tracking issue ## Why are the changes needed? Persisting

FlyteDirectory

doesn't work correctly when there are multiple files with the same name in the directory. Let's have two files: • /home/user/my_dir/file.txt • /home/user/my_dir/subdir/file.txt (with identical content in both files) And a trivial workflow:

@task()
def first_task(fd: fl.FlyteDirectory) -> None:
    fd.download()

@fl.workflow
def my_workflow(fd: fl.FlyteDirectory) -> None:
   first_task(fd)

Running with

pyflyte run ... --fd /home/user/my_dir

results in only

file.txt

being uploaded in the flyte S3 bucket. The file

subdir/file.txt

is skipped. This is of course not as expected. In addition, note that if the files have a different content, this raises an exception in

get_upload_signed_url

!

USER:EntityAlreadyExists: error=None, cause=<_InactiveRpcError of RPC that terminated with:
        status = StatusCode.ALREADY_EXISTS
        details = "file already exists at location [<s3://flyte/flytesnacks/development/ABSCCKRIMI35Z5XNGNQTO7FZIQ======/file.txt>], specify a matching hash if you wish to rewrite"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:6, grpc_message:"file already exists at location [<s3://flyte/flytesnacks/development/ABSCCKRIMI35Z5XNGNQTO7FZIQ======/file.txt>], specify a matching hash if you wish to rewrite"}"

Our current workaround is to pack a directory into an archive and use

FlyteFile

. ## What changes were proposed in this pull request? This bugfix is based on the observation that

get_upload_signed_url

is only provided file basename and file content hash to determine the URL. This leads to the both files ending up in the same location (the directory is flatten). To fix that, I add the path relative to the local root to the filename. This seems to work and both files are uploaded. Disclaimer: I have no idea if there are any side effects of this fix. It can be that some other functionality gets broken. Or there might be a better fix. I would appreciate a review from the maintainers:). ## How was this patch tested? Empirical test: For the minimal example above this works, the call to

super()._put

in

FlyteFS._put()

returns two objects:

['<s3://flyte/flytesnacks/development/QLGCX23Y7HX3FI2LIEGVF5OW7Q======/subdir/file.txt>', '<s3://flyte/flytesnacks/development/QLGCX23Y7HX3FI2LIEGVF5OW7Q======/file.txt>']

. This works both when the file content is identical and when it's different. ### Setup process ### Screenshots ## Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. ## Related PRs ## Docs link https://bito.ai#summarystart| ### Summary by Bito This pull request fixes a bug in the FlyteDirectory persistence mechanism that prevented the upload of multiple files with the same name and content. By including the relative path of files in the upload process, it enhances the Flyte framework's ability to manage directories with duplicate files effectively. flyteorg/flytekit