<#CNMKCU6FR|general> I am trying to pass a tensorf...
# announcementss
Sujith Samuel
09/06/2022, 8:59 AM#general I am trying to pass a tensorflow h5 model file between tasks but seems to have hit an issue. Is there any example for passing h5 files between tasks.
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[adpz8w6l67zrzfglb9m9-n4-0] terminated with exit code (137). Reason [Error]. Message:
use python type that flyte support."}
{"asctime": "2022-09-06 083946,752", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type <class 'keras.engine.training.Model'> found, Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend to use python type that flyte support."}
Different versions of python message is also confusing since all of my tasks are present in the same container and follow the same versions.
s
Shivay Lamba
09/06/2022, 9:14 AMHi Sujith, thanks for your question. can you confirm that the Python version for tasks running and the version of Python used when creating the tensorflow model is the same?
Also for example for h5 file, maybe @Samhita Alla can help
s
Sujith Samuel
09/06/2022, 9:25 AMOne of the tasks in the workflow is creating the model and the other task is using the model which is passed to it to upload to an S3 bucket. I assume since both the tasks are packaged into the same container, the versions would be same
s
Samhita Alla
09/06/2022, 9:42 AM
@Sujith Samuel, I don’t think pickling is the issue here since it’s always printed as a warning. Could you inspect the pod and let me know what’s happening in there by sending the log?
s
Sujith Samuel
09/06/2022, 11:42 AMOne of the decorators involved in the task was an mlflow decorator which was giving error. Once I cleared that, the error went away and the model was saved in the s3 location
Sujith Samuel
09/06/2022, 11:42 AMThanks a lot for your help @Samhita Alla @Shivay Lamba
k
Ketan (kumare3)
09/06/2022, 1:34 PM
@Sujith Samuel it's odd to pickle an h5 file
e
Eduardo Apolinario (eapolinario)
09/07/2022, 7:03 AM@Sujith Samuel, can you share the the skeleton for your workflow? Also, can you double-check which version of flytekit you're running?
s
Sujith Samuel
09/07/2022, 4:47 PM@Ketan (kumare3), yeah so i was basically passing the h5 file between tasks. and i guess flyte pickles and passes between tasks.
k
Ketan (kumare3)
09/08/2022, 5:37 AM
Flyte pickles if no transformers are available
Ketan (kumare3)
09/08/2022, 5:39 AM
So for example it maybe possible to serialize and provide a transformer for keras.Model so that it would automatically save it using h5
Ketan (kumare3)
09/08/2022, 5:39 AM
# Create and train a new model instance.
model = create_model()
model.fit(train_images, train_labels, epochs=5)
# Save the entire model as a SavedModel.
!mkdir -p saved_model
model.save('saved_model/my_model')
s
Samhita Alla
09/08/2022, 6:09 AM
@Sujith Samuel, if you’re up for a contribution, here’s the issue: https://github.com/flyteorg/flyte/issues/2570.
179 Views