Hello community, does anybody have any insight into modifying the gRPC requests that go on in the backend after a successfully authenticated

pyflyte run

command with an external auth server? I can see that it is part of the flyteIdl repository but have not worked with Go or gRPC in the past - though I am open to trying nonetheless. We are receiving 403 forbidden errors due to the

flyte-binary

pod/deployment being unable to send the

audience

parameter. I am assuming that this is between FlyteAdmin and FlytePropeller, though I could be wrong. So essentially, the flow is like this: • Localhost/client/machine sends

pyflyte run --remote

command to gRPC backend - AWS ALB w/ SSL/TLS • Auth request is successful for pyflyte/flytekit - using Auth0 as external auth server • Web console registers and then displays workflow with UNKNOWN status • No Pods that were requested in the pyflyte command are scheduled • Inspection of flyte-binary deployment w/ kubectl shows that a missing

audience

parameter is needed • Tons of requests with 403 - seems retry logic never stops - Necessitates killing deployment • Auth0 Logs show that all the requests fail due to audience Relevent Github issue is here with further logs/details: https://github.com/flyteorg/flyte/issues/3662 Any assistance in the matter would be greatly appreciated. We believe that this is the final step in getting Auth0 working with a flyte-binary deployment and would be more than glad to provide supporting documentation/code from forks if need be.

Yes you will have to configure flytepropeller client to have the same audience parameters, in tge helm chart

Thank you for the reply. Do you happen to have links to the relevant resources that highlight how to assign the audience parameter to flytepropeller in flyte-binary's Helm chart? I've been looking through flyte-core (not sure if it's a dependency of the flyte-binary chart? so not sure if we can do a sub-chart value override) and the flyte-binary Helm charts and can't seem to find a setting that reflects this for propeller - either in the templates or values Perhaps we are looking in the wrong places, but alternatively, if you happen to know where this setting might be set/affected (in

flyte-binary

) manually through

kubectl apply

or

kubectl patches

for the various resources? For example, we can override the configmap for auth for flyteadmin by pulling the manifest from EKS, parsing and patching certain blocks/values, then reapplying it, etc May have found solution, will close issue when finalized

cc @jeev / @Yee