<#CNMKCU6FR|> I am trying to pass a tensorflow h5 ...
# announcementss
stocky-notebook-88311
09/06/2022, 8:59 AM#CNMKCU6FR I am trying to pass a tensorflow h5 model file between tasks but seems to have hit an issue. Is there any example for passing h5 files between tasks.
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[adpz8w6l67zrzfglb9m9-n4-0] terminated with exit code (137). Reason [Error]. Message:
use python type that flyte support."}
{"asctime": "2022-09-06 083946,752", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type <class 'keras.engine.training.Model'> found, Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend to use python type that flyte support."}
Different versions of python message is also confusing since all of my tasks are present in the same container and follow the same versions.
v
victorious-park-53030
09/06/2022, 9:14 AMHi Sujith, thanks for your question. can you confirm that the Python version for tasks running and the version of Python used when creating the tensorflow model is the same?
Also for example for h5 file, maybe @tall-lock-23197 can help
s
stocky-notebook-88311
09/06/2022, 9:25 AMOne of the tasks in the workflow is creating the model and the other task is using the model which is passed to it to upload to an S3 bucket. I assume since both the tasks are packaged into the same container, the versions would be same
t
tall-lock-23197
09/06/2022, 9:42 AM
@stocky-notebook-88311, I don’t think pickling is the issue here since it’s always printed as a warning. Could you inspect the pod and let me know what’s happening in there by sending the log?
s
stocky-notebook-88311
09/06/2022, 11:42 AMOne of the decorators involved in the task was an mlflow decorator which was giving error. Once I cleared that, the error went away and the model was saved in the s3 location
stocky-notebook-88311
09/06/2022, 11:42 AMThanks a lot for your help @tall-lock-23197 @victorious-park-53030
👍 1
f
freezing-airport-6809
09/06/2022, 1:34 PM
@stocky-notebook-88311 it's odd to pickle an h5 file
h
high-accountant-32689
09/07/2022, 7:03 AM@stocky-notebook-88311, can you share the the skeleton for your workflow? Also, can you double-check which version of flytekit you're running?
s
stocky-notebook-88311
09/07/2022, 4:47 PM@freezing-airport-6809, yeah so i was basically passing the h5 file between tasks. and i guess flyte pickles and passes between tasks.
f
freezing-airport-6809
09/08/2022, 5:37 AM
Flyte pickles if no transformers are available
👍 1
freezing-airport-6809
09/08/2022, 5:39 AM
So for example it maybe possible to serialize and provide a transformer for keras.Model so that it would automatically save it using h5
freezing-airport-6809
09/08/2022, 5:39 AM
# Create and train a new model instance.
model = create_model()
model.fit(train_images, train_labels, epochs=5)
# Save the entire model as a SavedModel.
!mkdir -p saved_model
model.save('saved_model/my_model')
t
tall-lock-23197
09/08/2022, 6:09 AM
@stocky-notebook-88311, if you’re up for a contribution, here’s the issue: https://github.com/flyteorg/flyte/issues/2570.
183 Views