So I am doing this tutorial. https://github.com/unionai-oss/llm-fine-tuning/tree/main/flyte_llama#readme

pyflyte -c $FLYTECTL_CONFIG run --remote \                   
    --copy-all \
    --project $FLYTE_PROJECT \
    flyte_llama/workflows.py train_workflow \
    --config config/flyte_llama_7b_qlora_v0.json

The issue I am having is that, the docker build works, but the push never finishes. I am using gcp registry if that matters, but I should have a valid token. I am fairly certain this is a docker/envd issue not a pyflyte issue, but I haven't been able to diagnose why the push never finishes.

#21 DONE 201.0s
#22 bash -c set -euo pipefail
#22 DONE 0.6s
#23 exporting to image
#23 exporting layers
#23 exporting layers 203.5s done
#23 exporting manifest sha256:34901d58a06f61d4df5047b82a86d618cdda4a74d0f128b7dd10b03b395e52fd 0.0s done
#23 exporting config sha256:a36f0f5954ec5d6a30477e8ed35fc511bba7cdc3e4dcf58e6b96fdbd019c65f7 done
#23 pushing layers

I have tested on other ImageSpec tutorial, and I can push smaller images just fine.

ohh interesting

probably just my slow internet upload speed.