Hi there I ve just started installing flyte the binary on an

Question

Hi there I ve just started installing flyte the binary on an existing EKS cluster and have been walking through the getting started docs I m port forwarding from the flyte binary service on the cluster have created a test project and have created a local workflow script containing <https docs flyte org projects cookbook en latest getting started machine learning html|this example> However when I run `pyflyte run p testflyte remote example py training workflow hyperparameters C 0 1 ` I get a 403 error There s a signed url but this is rejected by the metadata bucket The cluster itself has a flyte service account with read write access to the bucket but the above is all happening locally Any ideas how I can debug this as I m currently a bit stumped Thanks a lot slightly smiling face

Niels Bantilan · Answer

< Yee> < jeev> any insight here

jeev · Answer

Can we get a paste of command and error please

Ed Fincham · Answer

The command ```pyflyte run p testflyte remote example py training workflow hyperparameters C 0 1 ``` The traceback ```Traceback most recent call last File home ed venv lib python3 10 site packages flytekit core data persistence py line 472 in put data DataPersistencePlugins find plugin remote path data config=self data config put File home ed venv lib python3 10 site packages flytekit extras persistence http py line 72 in put raise user FlyteValueException flytekit exceptions user FlyteValueException Value error Received 403 Request to send data lt BUCKET gt testflyte development OOXUOJ6GHCUQB2IUIQUAJ3257M 3D 3D 3D 3D 3D 3D scriptmode tar gz X Amz Algorithm=AWS4 HMAC SHA256 amp X Amz Credential= lt CREDENTIAL gt The above exception was the direct cause of the following exception Traceback most recent call last File home ed venv bin pyflyte line 8 in lt module gt sys exit main File home ed venv lib python3 10 site packages click core py line 1130 in call return self main args kwargs File home ed venv lib python3 10 site packages click core py line 1055 in main rv = self invoke ctx File home ed venv lib python3 10 site packages click core py line 1657 in invoke return process result sub ctx command invoke sub ctx File home ed venv lib python3 10 site packages click core py line 1657 in invoke return process result sub ctx command invoke sub ctx File home ed venv lib python3 10 site packages click core py line 1657 in invoke return process result sub ctx command invoke sub ctx File home ed venv lib python3 10 site packages click core py line 1404 in invoke return ctx invoke self callback ctx params File home ed venv lib python3 10 site packages click core py line 760 in invoke return callback args kwargs File home ed venv lib python3 10 site packages flytekit clis sdk in container run py line 552 in run remote entity = remote register script File home ed venv lib python3 10 site packages flytekit remote remote py line 800 in register script upload location md5 bytes = fast register single script File home ed venv lib python3 10 site packages flytekit tools script mode py line 112 in fast register single script flyte ctx file access put data archive fname upload location signed url File home ed venv lib python3 10 site packages flytekit core data persistence py line 476 in put data raise FlyteAssertion flytekit exceptions user FlyteAssertion Failed to put data from tmp tmppay0olxb script mode tar gz to lt BUCKET gt testflyte development OOXUOJ6GHCUQB2IUIQUAJ3257M 3D 3D 3D 3D 3D 3D scriptmode tar gz X Amz Algorithm=AWS4 HMAC SHA256 amp X Amz Credential= lt CREDENTIAL gt recursive=False Original exception Value error Received 403 Request to send data lt BUCKET gt testflyte development OOXUOJ6GHCUQB2IUIQUAJ3257M 3D 3D 3D 3D 3D 3D scriptmode tar gz X Amz Algorithm=AWS4 HMAC SHA256 amp X Amz Credential= lt CREDENTIAL gt failed ``` Thanks for getting back to me slightly smiling face

Niels Bantilan · Answer

looks like this has something to do with fast registration right looks like flytekit doesn t have to correct credentials to upload the tarfile to the configured s3 bucket

jeev · Answer

signed urls only require that the generator of the url has permissions iirc

jeev · Answer

then anyone with the url can use it

jeev · Answer

have we confirmed that flyteadmin s credentials can write to the bucket

Ed Fincham · Answer

So as I understand it pyflyte is trying to generate a signed url but it lacks the credentials hence the 403 I expect pyflyte needs some local aws creds but I couldn t find any documentation regarding setup

jeev · Answer

no flyte binary generates the signed url and passes to pyflyte to use

jeev · Answer

< Ed Fincham> is flyte binary getting credentials via an iam role

Ed Fincham · Answer

Yes

jeev · Answer

and IRSA is set up correctly I imagine

jeev · Answer

is the iam role able to write to the bucket

Ed Fincham · Answer

Ah My values might be wrong i created a service account separately but set `serviceAccount create` to true

jeev · Answer

hmm can you describe the flyte binary deployment and see if it s using the right KSA

jeev · Answer

and if so check if the iam role annotation is set correctly on that KSA

Ed Fincham · Answer

It s using the wrong one but I think this is probably enough to get me sorted I ll try tomorrow and let you know Thank you so much for the help slightly smiling face