https://flyte.org logo
#ask-the-community
Title
# ask-the-community
k

Klemens Kasseroller

01/15/2024, 10:31 AM
Hi, we are using FlyteFiles a lot in our workflows and sometimes we need to access the remote_source property of the FlyteFile. When running workflows locally, remote_source is None, which means we have to explicitly check "if remote_source is None -> use .path instead". To make local execution of flyte workflows be as close as possible to the remote execution, in my opinion it would be a better user experience to have remote_source locally contain the same value as .path. The same strategy is applied when calling .download() i.e. it returns the local path, instead of actually downloading something. I would add this to the list of FlyteFile UX improvements (https://github.com/flyteorg/flyte/issues/4542), but first would like to hear some opinions and if there are some technical implications. Thanks.
k

Ketan (kumare3)

01/15/2024, 4:03 PM
That should not be the case? It should have the remote surrender to a local path
Please add it to the list
k

Klemens Kasseroller

01/16/2024, 7:31 AM
I just verified again:
Copy code
from flytekit import workflow, task
from flytekit.types.file import FlyteFile

@task
def t1() -> FlyteFile:
    filename = "myfile.txt"
    with open(filename, "w") as f:
        f.write("ABC")
    return FlyteFile(filename)

@task
def t2(inp: FlyteFile) -> None:
    print("Remote source: ", inp.remote_source)
    print("Path: ", inp.path)
    print("Download: ", inp.download())


@workflow
def wf():
    out1 = t1()
    t2(inp=out1)


if __name__ == '__main__':
    wf()

Output:
Remote source:  None
Path:  /tmp/flyte-kucg8k7f/raw/88258869af6a3ac5e652f585207a45fe/myfile.txt
Download:  /tmp/flyte-kucg8k7f/raw/88258869af6a3ac5e652f585207a45fe/myfile.txt
Will add it to the list!
k

Ketan (kumare3)

01/17/2024, 4:14 AM
cc @Niels Bantilan is this something you did, where FlyteFile is only set on path?
n

Niels Bantilan

01/22/2024, 5:22 PM
Not sure, this PR may have caused this behavior: the intent was to use the original file/directory path on local executions instead of copying it over to a temporary location
2 Views