https://flyte.org
Join Slack
Did anyone run into issue with Flyte retrieving items from storage blob? [1/1] currentAttempt done. ...
# flyte-support
c
Did anyone run into issue with Flyte retrieving items from storage blob? [1/1] currentAttempt done. Last Error: USER::load_distribution(additional_distribution, dest_dir) │ │ │ │ /usr/local/lib/python3.10/site-packages/flytekit/core/utils.py:295 in │ │ wrapper │ │ │ │ ❱ 295 │ │ │ │ return func(*args, **kwargs) │ │ │ │ /usr/local/lib/python3.10/site-packages/flytekit/tools/fast_registration.py: │ │ 113 in download_distribution │ │ │ │ ❱ 113 │ FlyteContextManager.current_context().file_access.get_data(additio │ │ │ │ /usr/local/lib/python3.10/site-packages/flytekit/core/data_persistence.py:30 │ │ 1 in get_data │ │ │ │ ❱ 301 │ │ │ raise FlyteAssertion( │ ╰──────────────────────────────────────────────────────────────────────────────╯ FlyteAssertion: Failed to get data from gs://flyte-blob-storage/flytesnacks/development/MHEQCB3WRLH247GOTRY3CZQIWE====== /fastea66e2416c583fdc6995e9e030c414bf.tar.gz to /root/ (recursive=False). Original exception: 403, message='Forbidden', url=URL('https://storage.googleapis.com/download/storage/v1/b/flyte-blob-storage /o/flytesnacks%2Fdevelopment%2FMHEQCB3WRLH247GOTRY3CZQIWE======%2Ffastea66e2416c 583fdc6995e9e030c414bf.tar.gz?alt=media') The https link seems to include extra entries which results in 403 error
g
does you pod have permission to download the file?
c
SA has admin permissions on that storage bucket
but I think the issue is with the URL as it does not match the actual URL for the object showing via GUI
there is no /download/storage/v1/b/ in the path
g
have you changed container path?
Copy code
storage.yaml: |
    storage:
      type: s3
      container: "my-s3-bucket" <- this
you might have changed the default prefix somewhere in admin config map.
c
How should this be for gce?
here is my storage.yaml
Copy code
kubectl exec flyte-flyte-binary-75544db67f-rhwzp --namespace flyte -c flyte -- cat /etc/flyte/config.d/003-storage.yaml
propeller:
  rawoutput-prefix: <gs://flyte-blob-storage/data>
storage:
  type: stow
  stow:
    kind: google
    config:
      json: ""
      project_id: flyte-com-dev-per-gke
      scopes: <https://www.googleapis.com/auth/cloud-platform>
  container: flyte-blob-storage
g
do all the tasks fail with the same error?
cc @thankful-minister-83577
c
yup
I also tried to change the type to gcs but that did not change the type in config file
169 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by