https://flyte.org logo
#ask-the-community
Title
# ask-the-community
e

Ed Fincham

02/15/2023, 3:42 PM
Hi there, I've just started installing flyte (the binary) on an existing EKS cluster and have been walking through the getting started docs. I'm port-forwarding from the flyte-binary service on the cluster, have created a test project, and have created a local workflow script containing this example. However, when I run:
pyflyte run -p testflyte --remote example.py training_workflow --hyperparameters '{"C": 0.1}'
I get a 403 error. There's a signed url, but this is rejected by the metadata bucket. The cluster itself has a flyte service account with read/write access to the bucket, but the above is all happening locally. Any ideas how I can debug this as I'm currently a bit stumped! Thanks a lot 🙂
n

Niels Bantilan

02/15/2023, 4:18 PM
@Yee @jeev any insight here?
j

jeev

02/15/2023, 4:19 PM
Can we get a paste of command and error please?
e

Ed Fincham

02/15/2023, 6:25 PM
The command:
Copy code
pyflyte run -p testflyte --remote example.py training_workflow --hyperparameters '{"C": 0.1}'
The traceback:
Copy code
Traceback (most recent call last):
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 472, in put_data
    DataPersistencePlugins.find_plugin(remote_path)(data_config=self.data_config).put(
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/extras/persistence/http.py", line 72, in put
    raise user.FlyteValueException(
flytekit.exceptions.user.FlyteValueException: Value error!  Received: 403. Request to send data <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ed/venv/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/clis/sdk_in_container/run.py", line 552, in _run
    remote_entity = remote.register_script(
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/remote/remote.py", line 800, in register_script
    upload_location, md5_bytes = fast_register_single_script(
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/tools/script_mode.py", line 112, in fast_register_single_script
    flyte_ctx.file_access.put_data(archive_fname, upload_location.signed_url)
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 476, in put_data
    raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmppay0olxb/script_mode.tar.gz to <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL> (recursive=False)

Original exception: Value error!  Received: 403. Request to send data <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL> failed.
Thanks for getting back to me 🙂
n

Niels Bantilan

02/15/2023, 8:07 PM
looks like this has something to do with fast registration right? looks like flytekit doesn’t have to correct credentials to upload the tarfile to the configured s3 bucket
j

jeev

02/15/2023, 8:16 PM
signed urls only require that the generator of the url has permissions iirc
then anyone with the url can use it
have we confirmed that flyteadmin’s credentials can write to the bucket?
e

Ed Fincham

02/15/2023, 8:31 PM
So as I understand it, pyflyte is trying to generate a signed url but it lacks the credentials, hence the 403. I expect pyflyte needs some local aws creds, but I couldn't find any documentation regarding setup
j

jeev

02/15/2023, 8:31 PM
no flyte-binary generates the signed url and passes to pyflyte to use.
@Ed Fincham is flyte-binary getting credentials via an iam role?
e

Ed Fincham

02/15/2023, 8:33 PM
Yes
j

jeev

02/15/2023, 8:34 PM
and IRSA is set up correctly I imagine?
is the iam role able to write to the bucket?
e

Ed Fincham

02/15/2023, 8:35 PM
Ah. My values might be wrong - i created a service account separately but set
serviceAccount.create
to true
j

jeev

02/15/2023, 8:36 PM
hmm. can you describe the flyte-binary deployment and see if it’s using the right KSA?
and if so, check if the iam role annotation is set correctly on that KSA?
e

Ed Fincham

02/15/2023, 8:38 PM
It's using the wrong one but I think this is probably enough to get me sorted. I'll try tomorrow and let you know. Thank you so much for the help 🙂
44 Views