Hi there, I've just started installing flyte (the...
# ask-the-community
e
Hi there, I've just started installing flyte (the binary) on an existing EKS cluster and have been walking through the getting started docs. I'm port-forwarding from the flyte-binary service on the cluster, have created a test project, and have created a local workflow script containing this example. However, when I run:
pyflyte run -p testflyte --remote example.py training_workflow --hyperparameters '{"C": 0.1}'
I get a 403 error. There's a signed url, but this is rejected by the metadata bucket. The cluster itself has a flyte service account with read/write access to the bucket, but the above is all happening locally. Any ideas how I can debug this as I'm currently a bit stumped! Thanks a lot 🙂
n
@Yee @jeev any insight here?
j
Can we get a paste of command and error please?
e
The command:
Copy code
pyflyte run -p testflyte --remote example.py training_workflow --hyperparameters '{"C": 0.1}'
The traceback:
Copy code
Traceback (most recent call last):
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 472, in put_data
    DataPersistencePlugins.find_plugin(remote_path)(data_config=self.data_config).put(
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/extras/persistence/http.py", line 72, in put
    raise user.FlyteValueException(
flytekit.exceptions.user.FlyteValueException: Value error!  Received: 403. Request to send data <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ed/venv/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/clis/sdk_in_container/run.py", line 552, in _run
    remote_entity = remote.register_script(
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/remote/remote.py", line 800, in register_script
    upload_location, md5_bytes = fast_register_single_script(
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/tools/script_mode.py", line 112, in fast_register_single_script
    flyte_ctx.file_access.put_data(archive_fname, upload_location.signed_url)
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 476, in put_data
    raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmppay0olxb/script_mode.tar.gz to <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL> (recursive=False)

Original exception: Value error!  Received: 403. Request to send data <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL> failed.
Thanks for getting back to me 🙂
n
looks like this has something to do with fast registration right? looks like flytekit doesn’t have to correct credentials to upload the tarfile to the configured s3 bucket
j
signed urls only require that the generator of the url has permissions iirc
then anyone with the url can use it
have we confirmed that flyteadmin’s credentials can write to the bucket?
e
So as I understand it, pyflyte is trying to generate a signed url but it lacks the credentials, hence the 403. I expect pyflyte needs some local aws creds, but I couldn't find any documentation regarding setup
j
no flyte-binary generates the signed url and passes to pyflyte to use.
@Ed Fincham is flyte-binary getting credentials via an iam role?
e
Yes
j
and IRSA is set up correctly I imagine?
is the iam role able to write to the bucket?
e
Ah. My values might be wrong - i created a service account separately but set
serviceAccount.create
to true
j
hmm. can you describe the flyte-binary deployment and see if it’s using the right KSA?
and if so, check if the iam role annotation is set correctly on that KSA?
e
It's using the wrong one but I think this is probably enough to get me sorted. I'll try tomorrow and let you know. Thank you so much for the help 🙂
199 Views