https://flyte.org logo
#hacktoberfest-2022
Title
# hacktoberfest-2022
c

cryptic

10/28/2022, 6:40 AM
@Eduardo Apolinario (eapolinario) Would it be a proper way to serialize tensors using pickle?
s

Samhita Alla

10/28/2022, 7:07 AM
I don’t think so. Is there no serialization technique proposed by TensorFlow?
c

cryptic

10/28/2022, 7:08 AM
Yes there is but there is an issue here
to_python_value()
present in the TypeTransformer generally has only three arguments, but when using
tf.io.parse_tensor
I've to additionally pass
python_val
but most of the invocations in various files present in flytekit pass only three arguments, so this is giving me a TypeError when running pytest for one of my tests
s

Samhita Alla

10/28/2022, 7:14 AM
Can you share with me your PR?
c
I'll also be making the changes you'd proposed soon @Samhita Alla
s

Samhita Alla

10/28/2022, 7:30 AM
Let me see.
I’ve a few suggestions; let me get back to you in an hour.
c

cryptic

10/28/2022, 7:33 AM
the out_type is a mandatory argument for
tf.io.parse_tensor
@Samhita Alla There seems to be no other way of doing this
s

Samhita Alla

10/28/2022, 10:15 AM
1.
filename
should be
local_path
in
to_literal
2.
tf.io.read_file("tensor_data", name = None)
should be
tf.io.read_file(local_path)
3.
pathlib.Path(local_path).parent.mkdir(parents=True, exist_ok=True)
isn’t required because
tf.io.write_file
will always creae a directory recursively. 4. tf.io.parse_tensor(read_serial, out_type = python_val.dtype, name = None) 5. You can send the tensor
dtype
using `LiteralCollection`: https://github.com/flyteorg/flytekit/blob/7c915bb3a9347db09cc3cd2de60c53ddc5718d36/tests/flytekit/unit/core/test_literals_resolver.py#L79-L84. I took a stab at it: https://gist.github.com/samhita-alla/013601cd8b3a86a4277cdda54fa51a00.
@cryptic ^^^
Let me know if this still doesn’t work.
75 Views