Would it be a proper way

Question

< Eduardo Apolinario eapolinario > Would it be a proper way to serialize tensors using pickle

Samhita Alla · Answer

I don t think so Is there no serialization technique proposed by TensorFlow

cryptic · Answer

Yes there is but there is an issue here

cryptic · Answer

`to python value ` present in the TypeTransformer generally has only three arguments but when using `tf io parse tensor` I ve to additionally pass `python val` but most of the invocations in various files present in flytekit pass only three arguments so this is giving me a TypeError when running pytest for one of my tests

Samhita Alla · Answer

Can you share with me your PR

cryptic · Answer

This is slightly older <https github com flyteorg flytekit blob be865e5a97dc8f8095bd78ca3a73acb3ec5121b0 flytekit extras tensorflow tensor tensor transformer py|https github com flyteorg flytekit blob be865e5a97dc8f8095bd78ca3a73acb3ec5121b0 flytekit extras tensorflow tensor tensor transformer py> I ve have made few more changes locally

cryptic · Answer

I ll also be making the changes you d proposed soon < Samhita Alla>

cryptic · Answer

Here s the changes i d made <https flyte org slack com archives C043P7NDKL6 p1666881279621609 thread ts=1666848642 919639 amp cid=C043P7NDKL6|https flyte org slack com archives C043P7NDKL6 p1666881279621609 thread ts=1666848642 919639 amp cid=C043P7NDKL6>

Samhita Alla · Answer

Let me see

Samhita Alla · Answer

I ve a few suggestions let me get back to you in an hour

cryptic · Answer

the out type is a mandatory argument for `tf io parse tensor`

cryptic · Answer

< Samhita Alla> There seems to be no other way of doing this

Samhita Alla · Answer

1 `filename` should be `local path` in `to literal` 2 `tf io read file tensor data name = None ` should be `tf io read file local path ` 3 `pathlib Path local path parent mkdir parents=True exist ok=True ` isn t required because `tf io write file` will always creae a directory recursively 4 tf io parse tensor read serial out type = python val dtype name = None 5 You can send the tensor `dtype` using `LiteralCollection` I took a stab at it

Samhita Alla · Answer

< cryptic> ^^^

Samhita Alla · Answer

Let me know if this still doesn t work