curved-petabyte-84246
10/23/2023, 7:01 AMFailed with Unknown Exception <class 'TypeError'> Reason: Encountered error while executing workflow 'example.training_workflow':
Error encountered while executing 'training_workflow':
Failed to convert outputs of task 'example.get_data' at position 0:
[Errno 2] Failed to open local file '/var/folders/vs/v9fd0vyx6b735n5_l607sq440000gn/T/flyte-nl8jmq53/raw/2bb31bdcb6e0b0733d1bad9b87a3e886/00000'. Detail: [errno 2] No such file or directorySo I did a little digging. From the exception stacktrace it seems like the error happens when attempting to write the output. While debugging I found that indeed this folder isn't created. The temp folder
/var/folders/vs/v9fd0vyx6b735n5_l607sq440000gn/T/flyte-nl8jmq53/
does exist, but the raw
sub folder isn't. I kept debugging until I noticed several things (in reversed order π).
The method flytekit.core.data_persistence.FileAccessProvider.get_random_remote_path
- the default remote is expectedly LocalFileSystem
.
self._default_protocol
is "file"
but in that method, this following line returns `('file', 'local')`:
default_protocol = self._default_remote.protocol
This causes get_random_remote_path
to eventually return a path that starts with file:///...
.
Right after getting the random path, in flytekit.types.structured.basic_dfs.PandasToParquetEncodingHandler.encode
these lines happen:
if not ctx.file_access.is_remote(uri):
Path(uri).mkdir(parents=True, exist_ok=True)
is_remote
returns True
but apparently Path.mkdir
when operating on file://...
path, does NOTHING.
So the necessary folder hierarchy does not exist and the code fails.
Potential root causes:
1. fsspec
for some reason, returns ('file', 'local')
instead of file
.
2. Another option, that Flyte
code in flytekit.core.data_persistence.FileAccessProvider.get_random_remote_path
if type(default_protocol) == list:
default_protocol = default_protocol[0]
was supposed to test tuple
condition as well - this would have probably solved the issue
3. Another possiblity is my Mac... maybe Path
behaves weirdly on Maccurved-petabyte-84246
10/23/2023, 7:03 AMPath
understands the file://..
uri as a relative path, where file:
is a folder! si it essentially created the whole hierarchy under the current working directory!!!
~/src/ext/flyte/getting-started
βββ __pycache__
β βββ example.cpython-310.pyc
βββ example.py
βββ file:
βββ var
βββ folders
βββ vs
βββ v9fd0vyx6b735n5_l607sq440000gn
βββ T
βββ flyte-acgpkxgo
βββ raw
βββ e6a9aed58dcb402830573514c3b21d6a
curved-petabyte-84246
10/23/2023, 7:14 AMfsspec
changed the LocalFileSystem
implementation just recently add added the "local"
essentially converting the protocol
to a tuple. See here.
Interesting... I can create a PR against Flyte
checking if it's a tuple and taking the first argument. WDYT?thankful-minister-83577
thankful-minister-83577
curved-petabyte-84246
10/23/2023, 11:48 AMfsspec
problematic version was published just 2 days agocurved-petabyte-84246
10/23/2023, 11:48 AMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
curved-petabyte-84246
10/23/2023, 11:49 AMcurved-petabyte-84246
10/23/2023, 12:27 PMcurved-petabyte-84246
10/23/2023, 12:27 PMcurved-petabyte-84246
10/23/2023, 12:27 PMthankful-minister-83577
curved-petabyte-84246
10/23/2023, 12:34 PMflytekit
it seems like it mostly impacts that specific point so testing if the protocol is either a list or a tuple and then taking the first component should work.curved-petabyte-84246
10/23/2023, 12:35 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
curved-petabyte-84246
10/23/2023, 12:37 PMcurved-petabyte-84246
10/23/2023, 12:43 PMmake test
but nothing installs tensorflow. Am I missing something? I'm ran make setup
and then make test
curved-petabyte-84246
10/23/2023, 12:45 PMthankful-minister-83577
thankful-minister-83577
curved-petabyte-84246
10/23/2023, 1:19 PMfreezing-airport-6809
curved-petabyte-84246
10/24/2023, 8:18 PMfsspec
I believe you are good to go.curved-petabyte-84246
10/24/2023, 8:19 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
1.10.0.b1
strong-furniture-18784
10/25/2023, 5:39 AMcurved-petabyte-84246
10/25/2023, 5:56 AMpip install flytekit==1.10.1b0
strong-furniture-18784
10/25/2023, 5:57 AMcurved-petabyte-84246
10/25/2023, 5:57 AMpip install --upgrade flytekit
to get the latest versionstrong-furniture-18784
10/25/2023, 5:58 AMfreezing-airport-6809
curved-petabyte-84246
10/25/2023, 6:31 AMfsspec
version so you might not have fixed the issue. I will rebase on top of your changes and double checkcurved-petabyte-84246
10/25/2023, 6:32 AM"fsspec>=2023.3.0,<=2023.9.2"
thankful-minister-83577
thankful-minister-83577
curved-petabyte-84246
10/25/2023, 8:47 AMcurved-petabyte-84246
10/25/2023, 8:47 AMfsspec==2023.10.0
thankful-minister-83577
thankful-minister-83577
curved-petabyte-84246
10/25/2023, 8:48 AMthankful-minister-83577
curved-petabyte-84246
10/25/2023, 9:44 AMgreen-agency-30976
10/25/2023, 10:36 AMfsspec>=2023.3.0,<=2023.9.2
fixes it for now without having to get a beta version of flytekitcurved-petabyte-84246
10/25/2023, 10:50 AMcurved-petabyte-84246
10/30/2023, 8:55 AM# def test_transformer_to_literal_localss():
# random_dir = context_manager.FlyteContext.current_context().file_access.get_random_local_directory()
# fs = FileAccessProvider(local_sandbox_dir=random_dir, raw_output_prefix=os.path.join(random_dir, "raw"))
# ctx = context_manager.FlyteContext.current_context()
# with context_manager.FlyteContextManager.with_context(ctx.with_file_access(fs)) as ctx:
#
# tf = FlyteDirToMultipartBlobTransformer()
# lt = tf.get_literal_type(FlyteDirectory)
# # Can't use if it's not a directory
# with pytest.raises(FlyteAssertion):
# p = "/tmp/flyte/xyz"
# path = pathlib.Path(p)
# try:
# path.unlink()
# except OSError:
# ...
# with open(p, "w") as fh:
# fh.write("hello world\n")
# tf.to_literal(ctx, FlyteDirectory(p), FlyteDirectory, lt)
This test was failing before. There's another test with a similar name (without the typo)curved-petabyte-84246
10/30/2023, 8:55 AMfreezing-airport-6809