<#3313 [WIP] [Fix] Issue when using FlyteFile with...
# flyte-github
a
#3313 [WIP] [Fix] Issue when using FlyteFile with Elastic Pull request opened by machichima ## Tracking issue ## Why are the changes needed? Related to the discussion in the slack channel, when using
FlyteFile
with
Elastic
, the file cannot be downloaded and we will get warning: .venv/lib/python3.12/site-packages/flytekit/types/file/file.py356 RuntimeWarning: coroutine 'FileAccessProvider.async_get_data' was never awaited It seems like the
FlyteFile
does not play well with underlying multiprocessing spawn Based on my experiment, when we create the
_downloader
partial function, we use the context in main process: flytekit/flytekit/types/file/file.py Lines 321 to 326 in</flyteorg/flytekit/commit/eb5a67f76aaef96a44bde04afb72b87592cc8b7a|eb5a67f> | self._downloader = partial( | | ----------------------------------------------- | | ctx.file_access.get_data, | | ctx=ctx, | | remote_path=self._remote_source, # type: ignore | | local_path=self._local_path, | | ) | But when we use
Elastic
and run the
FlyteFile
within each process via spawn, the memory is not shared, but our
_downloader
function still use main process's context, which cannot be found in the sub-process. While the context required to convert the coroutine to a sync call (
loop_manager.synced
) isn’t present, the coroutine is returned as-is Below is the script for testing out the async in spawn with different context. The "Test 2" is which causing problem, and we can find that the things returned from the partial function is the
coroutine
(
Partial call is coroutine: {is_coro2}"
is
True
), which prove that somehow the
loop_manager.synced
doesn't work here. https://github.com/machichima/flytekit_test_scripts/blob/main/elastic-async-fsspec/compare_direct_vs_partial.py ## What changes were proposed in this pull request? Ensure that everytime we call
download
, we will get the
current_context
of the current process, to ensure calling the
get_data
synchronously. ## How was this patch tested? Test with following script: from flytekit import task, workflow, FlyteFile, ImageSpec, Resources from flytekitplugins.kfpytorch.task import Elastic, PyTorch DEFAULT_REMOTE_PATH = "s3://my-s3-bucket/test/test.txt" image_spec = ImageSpec( registry="localhost:30000", packages=[ "flytekitplugins-kfpytorch[elastic]", "numpy", "torch", ], ) @task( task_config=Elastic(nnodes=1, nproc_per_node=4), container_image=image_spec, requests=Resources(cpu="4", mem="4Gi"), limits=Resources(cpu="4", mem="4Gi"), ) def my_task(dataset: FlyteFile): path = dataset.download() print(f"path: {path}", flush=True) assert Path(path).is_file() # will pass now @workflow def wf(): file = FlyteFile(path=DEFAULT_REMOTE_PATH) outputs = my_task(dataset=file) ### Setup process ### Screenshots ## Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. ## Related PRs ## Docs link flyteorg/flytekit
c