Thread
#announcements
    Sujith Samuel

    Sujith Samuel

    3 weeks ago
    #general I am trying to pass a tensorflow h5 model file between tasks but seems to have hit an issue. Is there any example for passing h5 files between tasks. [1/1] currentAttempt done. Last Error: USER:😛od failed. No message received from kubernetes. [adpz8w6l67zrzfglb9m9-n4-0] terminated with exit code (137). Reason [Error]. Message: use python type that flyte support."} {"asctime": "2022-09-06 08:39:46,752", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type <class 'keras.engine.training.Model'> found, Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend to use python type that flyte support."} Different versions of python message is also confusing since all of my tasks are present in the same container and follow the same versions.
    Shivay Lamba

    Shivay Lamba

    3 weeks ago
    Hi Sujith, thanks for your question. can you confirm that the Python version for tasks running and the version of Python used when creating the tensorflow model is the same? Also for example for h5 file, maybe @Samhita Alla can help
    Sujith Samuel

    Sujith Samuel

    3 weeks ago
    One of the tasks in the workflow is creating the model and the other task is using the model which is passed to it to upload to an S3 bucket. I assume since both the tasks are packaged into the same container, the versions would be same
    Samhita Alla

    Samhita Alla

    3 weeks ago
    @Sujith Samuel, I don’t think pickling is the issue here since it’s always printed as a warning. Could you inspect the pod and let me know what’s happening in there by sending the log?
    Sujith Samuel

    Sujith Samuel

    3 weeks ago
    One of the decorators involved in the task was an mlflow decorator which was giving error. Once I cleared that, the error went away and the model was saved in the s3 location
    Thanks a lot for your help @Samhita Alla @Shivay Lamba
    Ketan (kumare3)

    Ketan (kumare3)

    3 weeks ago
    @Sujith Samuel it's odd to pickle an h5 file
    Eduardo Apolinario (eapolinario)

    Eduardo Apolinario (eapolinario)

    2 weeks ago
    @Sujith Samuel, can you share the the skeleton for your workflow? Also, can you double-check which version of flytekit you're running?
    Sujith Samuel

    Sujith Samuel

    2 weeks ago
    @Ketan (kumare3), yeah so i was basically passing the h5 file between tasks. and i guess flyte pickles and passes between tasks.
    Ketan (kumare3)

    Ketan (kumare3)

    2 weeks ago
    Flyte pickles if no transformers are available
    So for example it maybe possible to serialize and provide a transformer for keras.Model so that it would automatically save it using h5

    Create and train a new model instance.

    model = create_model() model.fit(train_images, train_labels, epochs=5)

    Save the entire model as a SavedModel.

    !mkdir -p saved_model model.save('saved_model/my_model')
    Samhita Alla

    Samhita Alla

    2 weeks ago
    @Sujith Samuel, if you’re up for a contribution, here’s the issue: https://github.com/flyteorg/flyte/issues/2570.