Hello. I'm new to Flyte, trying to test setup on local sandbox before attempting AWS deploy. Running WSL2 Ubuntu on Windows with Docker Desktop to produce a localhost remote cluster. What I don't quite understand is how files are stored. e.g. running this within a workflow:

# write to local path
out_path =  flytekit.current_context().working_directory
with open(os.path.join(out_path, 'test2.txt'), 'w') as f:
    f.write('blah')
        pth = f.name
return pth

If i run this locally (in Windows terminal) I get something like:

C:\Users\<user>\AppData\Local\Temp\flytel123\user_space123\test2.txt

If I run with --remote I get e.g.

/tmp/flyte-jp9gzohn/sandbox/local_flytekit/4bae375f282029a45f14f397429df2ff/test2.txt

I'd like to set up to use a folder (e.g. C:\Users\<user>\projectA\data and C:\Users\<user>\projectA\output) and run locally and then with --remote on the WSL2 hosted flytectl. Eventually will move to S3 but useful to have the local option for testing. C: is available in wsl under \mnt\c Ideas? Help?

hi @Peter Davidson firstly warmest welcome to the community. Thank you for joining

Running the code, localy from windows returns:

C:\Users\<user>\test.txt

Running it --remote in the sandbox returns:

"\wsl.localhost\docker-desktop-data\data\docker\volumes\aeb5ab0e1bf3f4e829dc928ef3e407044f1b11ab1978dbcda4689a2bcacae606_data\overlay2\6b40fb222b5a39e3c0e167965f289f398522078b4374c427d5b271b2493426f0\diff\root\test.txt"

The remote path is not very useful. How do you direct the output of flyte to a specific folder? Is this a flyteconfig setting or should I uzse FlyteFile or? Similarly, I would like to be able to point it to input files. --- Code:

import os
import flytekit
from flytekit import task, workflow


@task
def put_file() -> str:
    # write to local path
    out_path = os.path.expanduser('~')
    # out_path = flytekit.current_context().working_directory
    # out_path = '//root'
    with open(os.path.join(out_path, 'test.txt'), 'w') as f:
        f.write('blah')
        pth = f.name
    # with open(os.path.join(home, 'test.txt'), 'r') as f:
    #     pth = f.read()
    return pth


@workflow
def wf() -> str:
    return put_file()

You can send

--raw-data-prefix

to

pyflyte register

command to store the offloaded data.

Thanks. I need a bit more hand-holding. Say I run this:

pyflyte register flytest.py --raw-data-prefix C:\Users\<user>\flytest

What does this enable?

It’ll store the flyte offloaded data, i.e., the data returned from the flyte tasks in the given path.

thanks -> so how do I access this folder from within a workflow?

So when you pass a flytefile or a dataframe etc it is automatically stored and retrieved from the location in the background