cryptic
10/27/2022, 5:30 AMEduardo Apolinario (eapolinario)
10/27/2022, 5:40 AMipython==8.5.0
cryptic
10/27/2022, 5:47 AMEduardo Apolinario (eapolinario)
10/27/2022, 5:51 AMput_data
should take a path to the file you wrote, but right now it's pointing to the newly created directory.cryptic
10/27/2022, 5:52 AMEduardo Apolinario (eapolinario)
10/27/2022, 5:52 AMcryptic
10/27/2022, 5:53 AMEduardo Apolinario (eapolinario)
10/27/2022, 5:58 AMmake lint
pre-commit run --all-files
)cryptic
10/27/2022, 6:01 AMtf.io.write_file
doesn't support paths, so I did a os.chdir(str(local_path))
, but that didn't work out @Eduardo Apolinario (eapolinario)Eduardo Apolinario (eapolinario)
10/27/2022, 6:04 AMtf.io.write_file
with a path? What error do you see if you pass a path?cryptic
10/27/2022, 6:07 AMEduardo Apolinario (eapolinario)
10/27/2022, 6:07 AM- filename = "tensor_data"
+ filename = os.path.join(local_path, "tensor_data")
tf.io.write_file(filename, tf.io.serialize_tensor(python_val))
remote_path = ctx.file_access.get_random_remote_path(local_path)
- ctx.file_access.put_data(local_path, remote_path, is_multipart=False)
+ ctx.file_access.put_data(filename, remote_path, is_multipart=False)
cryptic
10/27/2022, 6:08 AMEduardo Apolinario (eapolinario)
10/27/2022, 6:14 AMcryptic
10/27/2022, 6:16 AMlv = <http://tf.to|tf.to>_literal(ctx, _python_val_, type(_python_val_), lt)
raises the error
tests/flytekit/unit/extras/tensorflow/test_transformations.py:56:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
flytekit/extras/tensorflow/tensor/tensor_transformer.py:58: in to_literal
ctx.file_access.put_data(local_path, remote_path, is_multipart=False)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <flytekit.core.data_persistence.FileAccessProvider object at 0x7fd612f9a520>
local_path = '/tmp/flyte-4tpz88yv/sandbox/local_flytekit/b7b927f373aed9b6b10c2e32f318c016'
remote_path = '/tmp/flyte-4tpz88yv/raw/ac09632c1597fcbf7684bad8698fc46f/b7b927f373aed9b6b10c2e32f318c016'
is_multipart = False
def put_data(self, local_path: Union[str, os.PathLike], remote_path: str, is_multipart=False):
"""
The implication here is that we're always going to put data to the remote location, so we .remote to ensure
we don't use the true local proxy if the remote path is a file://
:param Text local_path:
:param Text remote_path:
:param bool is_multipart:
"""
try:
with PerformanceTimer(f"Writing ({local_path} -> {remote_path})"):
DataPersistencePlugins.find_plugin(remote_path)(data_config=self.data_config).put(
local_path, remote_path, recursive=is_multipart
)
except Exception as ex:
> raise FlyteAssertion(
f"Failed to put data from {local_path} to {remote_path} (recursive={is_multipart}).\n\n"
f"Original exception: {str(ex)}"
) from ex
E flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/flyte-4tpz88yv/sandbox/local_flytekit/b7b927f373aed9b6b10c2e32f318c016 to /tmp/flyte-4tpz88yv/raw/ac09632c1597fcbf7684bad8698fc46f/b7b927f373aed9b6b10c2e32f318c016 (recursive=False).
E
E Original exception: [Errno 2] No such file or directory: '/tmp/flyte-4tpz88yv/sandbox/local_flytekit/b7b927f373aed9b6b10c2e32f318c016'
flytekit/core/data_persistence.py:455: FlyteAssertion
=================================== short test summary info ====================================FAILED tests/flytekit/unit/extras/tensorflow/test_transformations.py::test_to_python_value_and_literal[transformer0-Tensor-TensorflowTensor-python_val0]
E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/flyte-4tpz88yv/sandbox/local_flytekit/b7b927f373aed9b6b10c2e32f318c016'
/usr/lib/python3.8/shutil.py:261: FileNotFoundError
Eduardo Apolinario (eapolinario)
10/27/2022, 6:21 AM- / +
at the beginning of each row are from git diffcryptic
10/27/2022, 6:22 AM# Exclude setup.py to fix error: Duplicate module named "setup"
mypy plugins --exclude setup.py || true
flytekit-kf-mpi is not a valid Python package name
pre-commit run --all-files
flake8...................................................................Failed
- hook id: flake8
- exit code: 1
tests/flytekit/unit/extras/tensorflow/test_transformations.py:1:1: F401 'numpy as np' imported but unused
flytekit/extras/tensorflow/tensor/tensor_transformer.py:52:9: E265 block comment should start with '# '
black....................................................................Failed
- hook id: black
- files were modified by this hook
reformatted flytekit/extras/tensorflow/tensor/tensor_transformer.py
All done! ✨ 🍰 ✨
1 file reformatted, 525 files left unchanged.
isort....................................................................Passed
check yaml...............................................................Passed
fix end of files.........................................................Passed
trim trailing whitespace.................................................Passed
shellcheck...............................................................Failed
- hook id: shellcheck
- exit code: 1
Eduardo Apolinario (eapolinario)
10/27/2022, 2:26 PMcryptic
10/27/2022, 2:34 PMto_literal
of my tensor.py
global local_file_path
local_file_path = os.path.join(local_path, "tensor_data")
tf.io.write_file(local_file_path, tf.io.serialize_tensor(python_val))
remote_path = ctx.file_access.get_random_remote_path(local_file_path)
ctx.file_access.put_data(local_file_path, remote_path, is_multipart=False)
return Literal(scalar=Scalar(blob=Blob(metadata=meta, uri=remote_path)))
and I did
def to_python_value(self, ctx: FlyteContext, lv: Literal, python_val: T, expected_python_type: Type[T]) -> T:
try:
uri = lv.scalar.blob.uri
except AttributeError:
TypeTransformerFailedError(f"Cannot convert from {lv} to {expected_python_type}")
local_path = ctx.file_access.get_random_local_path()
ctx.file_access.get_data(uri, local_path, is_multipart=False)
#local_file_path = os.path.join(local_path, "tensor_data")
read_serial = tf.io.read_file(local_file_path, name=None)
return tf.io.parse_tensor(read_serial, out_type=python_val.dtype, name=None)
Eduardo Apolinario (eapolinario)
10/27/2022, 2:35 PMcryptic
10/27/2022, 2:38 PMEduardo Apolinario (eapolinario)
10/28/2022, 3:36 AMto_literal / to_python_value
are run in separate containers, so it doesn't make sense to use global variables to store state.
What error are you seeing in your test?
Also, keep in mind that we have several examples of type transformers that do something along the lines of what you're doing, so you could mimic exactly what they are doing.cryptic
10/28/2022, 3:50 AMto_python_value
pick a new random path? The error that I see is it is trying to load from a new random path in to_python_value
function and not from the one that was created by to_literal
/tmp/sandbox/flytekit/xhjdhiihghj/tensor_data
and in the next function it is trying to load from a different path eg. /tmp/sandbox/flytekit/cxzzdffghjjk/tensor_data
but cxzzdffghjjk
happens to be a file and not a folder and throws me the error "Not a file"