Hi there, I've just started installing flyte (the...
# flyte-support
s
Hi there, I've just started installing flyte (the binary) on an existing EKS cluster and have been walking through the getting started docs. I'm port-forwarding from the flyte-binary service on the cluster, have created a test project, and have created a local workflow script containing this example. However, when I run:
pyflyte run -p testflyte --remote example.py training_workflow --hyperparameters '{"C": 0.1}'
I get a 403 error. There's a signed url, but this is rejected by the metadata bucket. The cluster itself has a flyte service account with read/write access to the bucket, but the above is all happening locally. Any ideas how I can debug this as I'm currently a bit stumped! Thanks a lot 🙂
b
@thankful-minister-83577 @freezing-boots-56761 any insight here?
f
Can we get a paste of command and error please?
s
The command:
Copy code
pyflyte run -p testflyte --remote example.py training_workflow --hyperparameters '{"C": 0.1}'
The traceback:
Copy code
Traceback (most recent call last):
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 472, in put_data
    DataPersistencePlugins.find_plugin(remote_path)(data_config=self.data_config).put(
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/extras/persistence/http.py", line 72, in put
    raise user.FlyteValueException(
flytekit.exceptions.user.FlyteValueException: Value error!  Received: 403. Request to send data <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ed/venv/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/clis/sdk_in_container/run.py", line 552, in _run
    remote_entity = remote.register_script(
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/remote/remote.py", line 800, in register_script
    upload_location, md5_bytes = fast_register_single_script(
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/tools/script_mode.py", line 112, in fast_register_single_script
    flyte_ctx.file_access.put_data(archive_fname, upload_location.signed_url)
  File "/home/ed/venv/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 476, in put_data
    raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmppay0olxb/script_mode.tar.gz to <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL> (recursive=False)

Original exception: Value error!  Received: 403. Request to send data <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL> failed.
Thanks for getting back to me 🙂
b
looks like this has something to do with fast registration right? looks like flytekit doesn’t have to correct credentials to upload the tarfile to the configured s3 bucket
f
signed urls only require that the generator of the url has permissions iirc
☝️ 1
then anyone with the url can use it
have we confirmed that flyteadmin’s credentials can write to the bucket?
s
So as I understand it, pyflyte is trying to generate a signed url but it lacks the credentials, hence the 403. I expect pyflyte needs some local aws creds, but I couldn't find any documentation regarding setup
f
no flyte-binary generates the signed url and passes to pyflyte to use.
@silly-refrigerator-21057 is flyte-binary getting credentials via an iam role?
s
Yes
👍 1
f
and IRSA is set up correctly I imagine?
is the iam role able to write to the bucket?
👍 1
s
Ah. My values might be wrong - i created a service account separately but set
serviceAccount.create
to true
f
hmm. can you describe the flyte-binary deployment and see if it’s using the right KSA?
👍 1
and if so, check if the iam role annotation is set correctly on that KSA?
s
It's using the wrong one but I think this is probably enough to get me sorted. I'll try tomorrow and let you know. Thank you so much for the help 🙂
👍 3
308 Views