Hi there, I've just started installing flyte (the binary) on an existing EKS cluster and have been walking through the getting started docs. I'm port-forwarding from the flyte-binary service on the cluster, have created a test project, and have created a local workflow script containing this example. However, when I run:

pyflyte run -p testflyte --remote example.py training_workflow --hyperparameters '{"C": 0.1}'

I get a 403 error. There's a signed url, but this is rejected by the metadata bucket. The cluster itself has a flyte service account with read/write access to the bucket, but the above is all happening locally. Any ideas how I can debug this as I'm currently a bit stumped! Thanks a lot 🙂

@Yee @jeev any insight here?

Can we get a paste of command and error please?

The command:

pyflyte run -p testflyte --remote example.py training_workflow --hyperparameters '{"C": 0.1}'

The traceback:

Traceback (most recent call last):
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 472, in put_data
    DataPersistencePlugins.find_plugin(remote_path)(data_config=self.data_config).put(
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/extras/persistence/http.py", line 72, in put
    raise user.FlyteValueException(
flytekit.exceptions.user.FlyteValueException: Value error!  Received: 403. Request to send data <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ed/venv/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/clis/sdk_in_container/run.py", line 552, in _run
    remote_entity = remote.register_script(
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/remote/remote.py", line 800, in register_script
    upload_location, md5_bytes = fast_register_single_script(
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/tools/script_mode.py", line 112, in fast_register_single_script
    flyte_ctx.file_access.put_data(archive_fname, upload_location.signed_url)
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 476, in put_data
    raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmppay0olxb/script_mode.tar.gz to <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL> (recursive=False)

Original exception: Value error!  Received: 403. Request to send data <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL> failed.

Thanks for getting back to me 🙂

looks like this has something to do with fast registration right? looks like flytekit doesn’t have to correct credentials to upload the tarfile to the configured s3 bucket

signed urls only require that the generator of the url has permissions iirc

So as I understand it, pyflyte is trying to generate a signed url but it lacks the credentials, hence the 403. I expect pyflyte needs some local aws creds, but I couldn't find any documentation regarding setup

no flyte-binary generates the signed url and passes to pyflyte to use.

Yes

and IRSA is set up correctly I imagine?

Ah. My values might be wrong - i created a service account separately but set

serviceAccount.create

to true

hmm. can you describe the flyte-binary deployment and see if it’s using the right KSA?

It's using the wrong one but I think this is probably enough to get me sorted. I'll try tomorrow and let you know. Thank you so much for the help 🙂