https://flyte.org
Join Slack
Hi Team, I have noticed when users are logged in to Flyte Console via Auth0, any workflows launched ...
# flyte-deployment
q
Hi Team, I have noticed when users are logged in to Flyte Console via Auth0, any workflows launched from Console go to an 
UNKNOWN
state. I was wondering what could be possible cause for this? Do we need to inform any service other than Flyte Admin about the Auth? For Example Flyte Propeller? CC: @icy-agent-73298
t
hi
can you screenshare?
@quaint-byte-23550
q
Hi, Yee, I am not at my desk ATM, but I can try coming back in tomorrow if that works. I was just wondering in case someone noticed this
t
got it okay, let’s chat when you have a chance
f
I think propeller is not able to talk to admin
So it must be failing
Did you create and app Id and add it to propeller
q
I did not do that, I wonder if that needs to be added to documentation. I will double check. By app ID you mean Auth0 app client ID ?
f
yes - every identity provider has it
q
After setting clientId and properly supplying client Secret. I am getting the same issue of
Copy code
rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 403 Forbidden\nResponse: {\"error\":\"access_denied\",\"error_description\":\"No audience parameter was provided, and no default audience has been configured\"
So just like @icy-agent-73298 helped with Audience being set within 
FlyteAdmin
in the image:
Copy code
repository: <http://ghcr.io/flyteorg/flyteadmin|ghcr.io/flyteorg/flyteadmin>
    tag: v1.1.37-deviceauth
We need Audience variations of these images for scheduler and propeller. Please correct me if I am wrong. CC: @thankful-minister-83577 @nutritious-london-39005 @icy-agent-73298
t
can you try with flytectl?
and the admin logs please… they may have more info, help us narrow down where in the code the error is coming from
q
can you try with flytectl?
I just confirmed with my team that our Auth0 server does require an 
Audience
even from 
clientSecret
flow. So, I believe to make this work with the Auth server we do need Audience properly set.
t
that is a little weird.
@high-park-82026 will need help
i might be doing something wrong then… cuz it appears as if the golang std lib for client credentials doesn’t work with auth0.
t
@thankful-minister-83577 I believe audience can be passed in if it is set in the tokenSource’s Config’s EndpointParams. See https://github.com/golang/oauth2/commit/01b79d94479495d23e14e5ddc8f85a9271aff260
👍 1
q
Thanks @thankful-dress-89577 for pointing that out, I created this issue: https://github.com/flyteorg/flyte/issues/2959 Also added above suggestion in it. @thankful-minister-83577 @high-park-82026 @icy-agent-73298
t
Might be that the only change needed is here? https://github.com/flyteorg/flyteidl/blob/1fb8cf18b290ff7c65b860f2f4423a1ae99e3d45/clients/go/admin/token_source_provider.go#L155-L175 Would then need to propagate flyteidl update to other components.
h
@thankful-dress-89577 so the problem is, as Yee pointed out, that the underlying golang library we use: https://cs.opensource.google/go/x/oauth2/+/b44042a4:clientcredentials/clientcredentials.go Doesn’t set audience on the request to issue tokens from IdP
the request sent to issue a token doesn’t include Audience (not even as an optional field)… it seems to me that Auth0's implementation doesn’t follow the standard
t
I think I follow, but the linked commit above shows how oauth2 library did add support for setting audience in the request via the EndpointParams.
It isn’t a named “audience” argument, but EndpointParams is flexible enough to support it.
h
aha… I missed that part… is that in a released version of oauth2?
t
it was merged Mar 13, 2017, so I hope so 🙂
h
If you want to modify our code to populate that, I’m all for it! Here is what I think needs to happen: 1. This PR needs to go in to create an audience field in Admin’s metadata response. 2. We need to follow up with a PR in admin to populate it here 3. Follow up PR in flyteidl to consume it here (see how scopes and tokenUrl is done)
👍 2
t
do we also want to add an audience field to the config object?
q
Yes, config for Audience also needs to be provided from every service that uses ClientSecret. AFAIK these services are 
propeller
, 
scheduler
, and 
flytectl
.
t
oh i think not… just saw the deprecated comment.
it’s needed yeah @quaint-byte-23550 but it will discover it dynamically from admin.
q
Thanks to Prafulla for implementing the changes required in 
flyteadmin
and 
flyteidl
for providing Audience. I was successfully able to create flytescheduler and propeller images. After I supply audience value, propeller is created fine. But scheduler is continuously in PodInitialization, unable to debug it further I have supplied the values to audience in my config maps like:
Copy code
admin.yaml: |
    admin:
      clientId: <clientId>
      clientSecretLocation: /etc/secrets/client_secret
      audience: <audience_value>
      endpoint: flyteadmin:81
      insecure: true
CC: @thankful-minister-83577 please let me know what could I be doing wrong
t
what’s the error message?
q
All I see is that 
flytescheduler-check
is in CrashLoopBackOff state:
Copy code
terminated
Reason: Error - exit code: 2
Started at: 2022-10-12T14:39:43-04:00
Finished at: 2022-10-12T14:39:44-04:00
t
can you get the logs?
i’m not caught up with the changes. what was made?
q
Also, I checked admin logs and looks like Auth is working, as I see warnings related to the task only. So no 
Audience
error anymore.
t
i’m a little confused.
are we still doing the original plan? use the client credentials flow with auth0?
👍 1
did someone make the change to make use of the new audience field in the client credentials flow in the admin client?
q
Yes sticking to the original plan. Prafulla made these PRs for me to test: https://github.com/flyteorg/flyteadmin/pull/485 https://github.com/flyteorg/flyteidl/pull/329
t
i am not seeing that.
that pr doesn’t touch the client credentials flow
q
I dont understand, what do you not see? I think this PR: https://github.com/flyteorg/flyteidl/pull/329/files does change 
token_source_provider
file to return 
EndPointParams
. Which are then going to be used to return Audience parameter.
t
unf there’s still some work to do i think.
but how is that audience parameter passed to auth0
that still needs to be done.
q
Copy code
audienceValue := cfg.Audience
		if len(audienceValue) == 0 {
			audienceValue = clientMetadata.Audience
		}

		tokenProvider, err = NewClientCredentialsTokenSourceProvider(ctx, cfg, scopes, tokenURL, audienceValue)
t
oh this pr sorry i was looking at #314
q
I think that is what above PR is doing, getting Audience from the config, converting it to an 
EndPointParams
to be passed.
Yes, I shared 2 new PR links which does what is said.
t
and you pulled this pr into your propeller build?
and propeller is able to send eventing back to admin?
q
Yes, I pulled these PRs to build new 
flyteadmin
flytepropeller
flytescheduler
t
did you kick off a run? i dont’ think there’s traffic from propeller to admin until something is being run
q
yes, I did. Workflow goes into unknown state...
t
so auth is probably failing there too
q
I see propeller has same
Copy code
No audience parameter was provided, and no default audience has been configured
t
can you confirm first that the new client works? like just run the client as a unit test with golang and plug in the audience and secret/id
q
I ll try that out and try to debug. Thanks!
t
can you also confirm that the audience is being returned by the admin endpoint?
q
How can I access this endpoint from k8s deployment?
t
how do you access your flyte UI?
the flyte console
q
yes
t
yes it does?
q
I have console URL that I access the UI through
Yup, that endpoint is returning a valid audience value configured by me.
t
cool
then let’s confirm that a go unit test that hits auth0 also works
and then we can sort out what’s in between that’s breaking
q
So updating this thread. Thanks to @icy-agent-73298 again for coming through and adding PRs for new flyteidl and flyteadmin supporting audience in clients credential workflows. We were able to successfully run workflows after authentication on Flyte on Monday.
💯 1
❤️ 2
f
@quaint-byte-23550 this is awesome news
q
Thanks to team Flyte for their continuous support!
f
Thank you @quaint-byte-23550
Please help us spread awareness
m
Hello there, anyway to share how to setup auth0 with Flyte (maybe which version we need to use as well?) We'll need to do an equivalent setup soon
f
@mammoth-church-8767 it should work, but no docs
That are specific
m
Ok, I’m trying to setup it and got the same error on the propeller:
Copy code
No audience parameter was provided, and no default audience has been configured
and this on the admin side:
Copy code
{"json":{"src":"cookie.go:88"},"level":"debug","msg":"Existing [flyte_idt] cookie found","ts":"2022-12-20T14:30:01Z"}
{"json":{"src":"cookie.go:88"},"level":"debug","msg":"Existing [flyte_at] cookie found","ts":"2022-12-20T14:30:01Z"}
{"json":{"src":"cookie.go:88"},"level":"debug","msg":"Existing [flyte_rt] cookie found","ts":"2022-12-20T14:30:01Z"}
{"json":{"src":"cookie.go:88"},"level":"debug","msg":"Existing [flyte_user_info] cookie found","ts":"2022-12-20T14:30:01Z"}
{"json":{"src":"handlers.go:235"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-12-20T14:30:01Z"}
{"json":{"src":"token.go:83"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-12-20T14:30:01Z"}
{"json":{"src":"handlers.go:245"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-12-20T14:30:01Z"}
I’m using the 
v1.2.1
images and added the audience conf as below
Copy code
thirdPartyConfig:
          flyteClient:
            clientId: clientId
            redirectUri: <https://xxxx.auth0.com/callback>
            scopes:
            - offline
            - all
            audience: <https://xxxx.auth0.com/api/v2/>
And the audience is correctly return by is correctly return by the admin endpoint 
/config/v1/flyte_client
I’m not sure what I’m doing wrong there
f
Cc @icy-agent-73298 do you know off hand
i
We haven’t merged this PR https://github.com/flyteorg/flyteidl/pull/329/files and isn’t part of the release yet. Let me prioritize the reviews on it and get it checkedin and will require followup PR’s in flytepropeller to consume it. @mammoth-church-8767 if you want try it out yourself then you can use the idl changes directly in your propeller and build it which i can help you with it or else i will give you released version once the code is checkedin
m
Hooo I see apologized I only looked at the first PR 😅 and didn’t realized it wasn’t merged. Yeah, I’ll try to build it on my side to test it
i
np. Let me know if you run into issues. Meanwhile will try to get this checkedin
👌 1
m
Hey there, so I’m trying to build a release for flyteidl but got issues such as:
Copy code
Error: unknown command "RedisClient" for "mockery"
Error: unknown command "HandlerFactory" for "mockery"
I’m not a go dev, so it surely is an issue on my setup 😅
f
Seems like it - mocks
i
Hi @mammoth-church-8767, you dont need to create a release and you can directly consume this by building a new propeller image with the changes. In flytepropeller repo, you can run • 
go get <http://github.com/flyteorg/flyteidl|github.com/flyteorg/flyteidl>
<http://github.com/flyteorg/flyteidl@956c7a259b50a6607aa82a9d3cc0bcbe51919f84|github.com/flyteorg/flyteidl@956c7a259b50a6607aa82a9d3cc0bcbe51919f84>
which uses the commit from the PR • 
go mod tidy
• 
make docker_build
. this will build the propeller image with the idl changes • Push the built image to your local image repo and use the same in your flyte deployment. With the change you can now configure flytepropeller to send the audience field when having flyteadmin relay auth with auth0 provider. In your propeller config map you should be able add the audience key and value https://github.com/flyteorg/flyteidl/pull/329/files#diff-91f1e2cdbc64e0a780abe3c3eddfeb4bc61a1e099e7393a1331ff7f27be30c5cR56 in the admin section
m
Hey @icy-agent-73298 Happy new year! I finally been able to test it out. It works like a charm now using the Console or API with a provided bearer 🙂 As we want to setup a CI and via the Python SDK, I think we would need to update this too correct? (We are facing same issue about the audience) I’ll update the thread once all good
i
good to hear that it worked for you . yes to use flytekit a corresponding change would be needed there too. Would you be willing to make that contribution to the sdk which i suppose shouldn’t be much and also since you have an environment where you have auth setup with a non-default audience then it would be easily testable too. Let us know what you think
m
With pleasure, just make it worked 😉 so I’ll try to create a PR by the end of the week. But it will require your PRs to be merge before no for flyteidl? (I built it locally for now)
i
Awesome. yeah. will try to get those flyteidl ones by then . Thanks Jonathan.
f
Can I know more about this. I am working on making the sdk auth layer much better
i
@freezing-airport-6809 in the above case, the auth server is configured with no default audience and the client needs to have this audience configuration when requesting a token from the auth server and using that in the flyteadmin resource server for requesting resources. flyteidl changes have this changes and needs to go in along with flytekit changes to use this config.
236 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by