Guy Arad
10/23/2023, 7:01 AMFailed with Unknown Exception <class 'TypeError'> Reason: Encountered error while executing workflow 'example.training_workflow':
Error encountered while executing 'training_workflow':
Failed to convert outputs of task 'example.get_data' at position 0:
[Errno 2] Failed to open local file '/var/folders/vs/v9fd0vyx6b735n5_l607sq440000gn/T/flyte-nl8jmq53/raw/2bb31bdcb6e0b0733d1bad9b87a3e886/00000'. Detail: [errno 2] No such file or directorySo I did a little digging. From the exception stacktrace it seems like the error happens when attempting to write the output. While debugging I found that indeed this folder isn't created. The temp folder
/var/folders/vs/v9fd0vyx6b735n5_l607sq440000gn/T/flyte-nl8jmq53/
does exist, but the raw
sub folder isn't. I kept debugging until I noticed several things (in reversed order π).
The method flytekit.core.data_persistence.FileAccessProvider.get_random_remote_path
- the default remote is expectedly LocalFileSystem
.
self._default_protocol
is "file"
but in that method, this following line returns `('file', 'local')`:
default_protocol = self._default_remote.protocol
This causes get_random_remote_path
to eventually return a path that starts with file:///...
.
Right after getting the random path, in flytekit.types.structured.basic_dfs.PandasToParquetEncodingHandler.encode
these lines happen:
if not ctx.file_access.is_remote(uri):
Path(uri).mkdir(parents=True, exist_ok=True)
is_remote
returns True
but apparently Path.mkdir
when operating on file://...
path, does NOTHING.
So the necessary folder hierarchy does not exist and the code fails.
Potential root causes:
1. fsspec
for some reason, returns ('file', 'local')
instead of file
.
2. Another option, that Flyte
code in flytekit.core.data_persistence.FileAccessProvider.get_random_remote_path
if type(default_protocol) == list:
default_protocol = default_protocol[0]
was supposed to test tuple
condition as well - this would have probably solved the issue
3. Another possiblity is my Mac... maybe Path
behaves weirdly on MacPath
understands the file://..
uri as a relative path, where file:
is a folder! si it essentially created the whole hierarchy under the current working directory!!!
~/src/ext/flyte/getting-started
βββ __pycache__
β βββ example.cpython-310.pyc
βββ example.py
βββ file:
βββ var
βββ folders
βββ vs
βββ v9fd0vyx6b735n5_l607sq440000gn
βββ T
βββ flyte-acgpkxgo
βββ raw
βββ e6a9aed58dcb402830573514c3b21d6a
fsspec
changed the LocalFileSystem
implementation just recently add added the "local"
essentially converting the protocol
to a tuple. See here.
Interesting... I can create a PR against Flyte
checking if it's a tuple and taking the first argument. WDYT?Yee
Guy Arad
10/23/2023, 11:48 AMfsspec
problematic version was published just 2 days agoYee
Guy Arad
10/23/2023, 11:49 AMYee
Guy Arad
10/23/2023, 12:34 PMflytekit
it seems like it mostly impacts that specific point so testing if the protocol is either a list or a tuple and then taking the first component should work.Yee
Guy Arad
10/23/2023, 12:37 PMmake test
but nothing installs tensorflow. Am I missing something? I'm ran make setup
and then make test
Yee
Guy Arad
10/23/2023, 1:19 PMKetan (kumare3)
Guy Arad
10/24/2023, 8:18 PMfsspec
I believe you are good to go.Ketan (kumare3)
1.10.0.b1
Adhi Setiawan
10/25/2023, 5:39 AMGuy Arad
10/25/2023, 5:56 AMpip install flytekit==1.10.1b0
Adhi Setiawan
10/25/2023, 5:57 AMGuy Arad
10/25/2023, 5:57 AMpip install --upgrade flytekit
to get the latest versionAdhi Setiawan
10/25/2023, 5:58 AMKetan (kumare3)
Guy Arad
10/25/2023, 6:31 AMfsspec
version so you might not have fixed the issue. I will rebase on top of your changes and double check"fsspec>=2023.3.0,<=2023.9.2"
Yee
Guy Arad
10/25/2023, 8:47 AMfsspec==2023.10.0
Yee
Guy Arad
10/25/2023, 8:48 AMYee
Guy Arad
10/25/2023, 9:44 AMGauthier Castro
10/25/2023, 10:36 AMfsspec>=2023.3.0,<=2023.9.2
fixes it for now without having to get a beta version of flytekitGuy Arad
10/25/2023, 10:50 AM# def test_transformer_to_literal_localss():
# random_dir = context_manager.FlyteContext.current_context().file_access.get_random_local_directory()
# fs = FileAccessProvider(local_sandbox_dir=random_dir, raw_output_prefix=os.path.join(random_dir, "raw"))
# ctx = context_manager.FlyteContext.current_context()
# with context_manager.FlyteContextManager.with_context(ctx.with_file_access(fs)) as ctx:
#
# tf = FlyteDirToMultipartBlobTransformer()
# lt = tf.get_literal_type(FlyteDirectory)
# # Can't use if it's not a directory
# with pytest.raises(FlyteAssertion):
# p = "/tmp/flyte/xyz"
# path = pathlib.Path(p)
# try:
# path.unlink()
# except OSError:
# ...
# with open(p, "w") as fh:
# fh.write("hello world\n")
# tf.to_literal(ctx, FlyteDirectory(p), FlyteDirectory, lt)
This test was failing before. There's another test with a similar name (without the typo)Ketan (kumare3)