Also @Niels Bantilan, regarding https://github.com/flyteorg/flytekit/pull/1240, are there any use cases you could think of?

not sure I understand the question… I assumed

.tfrecord

is a TF-specific file format that we wanted to support out-of-the-box?

This PR adds a

tf.Train.Example

type and it internally stores the data in a tfrecord file for serialization. I’ve been wondering why someone might use

tf.Train.Example

as a data type.

@Samhita Alla This is the pattern tf docs https://www.tensorflow.org/tutorials/load_data/tfrecord suggest for serialising to

.tfrecord

unless i misunderstood the requirements of the original issue. The main use case(s) seems to be for image data and training on TPUs (see next comment in thread). The alternative i guess is to provide support for a

tf.train.Features

type which then gets converted to

Example

type and serialised and stored as

.tfrecord

- see the images i/o example at the end of https://www.tensorflow.org/tutorials/load_data/tfrecord

@Ryan Nazareth, makes sense. I have a couple of suggestions: • There’s no

tf.data

<-> tfrecord conversion available. We may need to add support for

tf.data.Dataset

as well. • IMO, this Flyte type should only perform conversion from `tf.Train.Example`/`tf.data` to tfrecord file (like Flyte ONNX types) cause tfrecord format per se can be used while training the model, but not the vice versa. We can also have a task to read the tfrecord file but it can include a lot of customizations, e.g., https://www.tensorflow.org/tutorials/load_data/tfrecord#read_the_tfrecord_file & https://keras.io/examples/keras_recipes/creating_tfrecords/#train-a-simple-model-using-the-generated-tfrecords. So I’m not sure if we want a task to read the tfrecord file. cc: @Niels Bantilan @Ketan (kumare3)

@Samhita Alla sure i can make the change and add support for

tf.data

. Also, regarding your second point, Im assuming in that case i should create a new flytefile type

TfRecordFile=FlyteFile[typing.TypeVar('tfrecord')]

which would then be returned as `TfRecordFile(path=lv.scalar.blob.uri)`in

def to_python_value(self, ...)

method of TypeTransformer, similar to the pytorch ONXX implementation ? Let me know once @Niels Bantilan and @Ketan (kumare3) have confirmed.

So after discussing with @Eduardo Apolinario (eapolinario) and thinking about this a little more, I think we still need to discuss the core question of: how do people actually use

tf.train.Example

, `TFRecord`s and

tf.data.Dataset

in real life, and how do we create reasonable i/o and serialization/deserialization boundaries between these objects, again, in the context of how people actual use them, and in a way that provides value to users? I created a draft proposal on the PR: https://github.com/flyteorg/flytekit/pull/1240#issuecomment-1292623521

Thanks, @Niels Bantilan! Added a suggestion.

Related to this discussion, I've been thinking about how to add support for

tf.data.Dataset

to Flyte (and whether it's even a good idea) and created this feature request issue: https://github.com/flyteorg/flyte/issues/3038