https://flyte.org logo
Powered by
s

Shahwar Saleem

09/28/2022, 9:09 PM
Hi Team, I have noticed when users are logged in to Flyte Console via Auth0, any workflows launched from Console go to an 
UNKNOWN
state. I was wondering what could be possible cause for this? Do we need to inform any service other than Flyte Admin about the Auth? For Example Flyte Propeller? CC: @Prafulla Mahindrakar
y

Yee

09/28/2022, 9:10 PM
hi
can you screenshare?
@Shahwar Saleem
s

Shahwar Saleem

09/28/2022, 9:15 PM
Hi, Yee, I am not at my desk ATM, but I can try coming back in tomorrow if that works. I was just wondering in case someone noticed this
y

Yee

09/28/2022, 9:15 PM
got it okay, let’s chat when you have a chance
k

Ketan (kumare3)

09/29/2022, 12:48 AM
I think propeller is not able to talk to admin
So it must be failing
Did you create and app Id and add it to propeller
s

Shahwar Saleem

09/29/2022, 2:20 PM
I did not do that, I wonder if that needs to be added to documentation. I will double check. By app ID you mean Auth0 app client ID ?
k

Ketan (kumare3)

09/29/2022, 8:27 PM
yes - every identity provider has it
y

Yee

10/05/2022, 6:27 PM
can you try with flytectl?
and the admin logs please… they may have more info, help us narrow down where in the code the error is coming from
s

Shahwar Saleem

10/05/2022, 6:32 PM
can you try with flytectl?
I just confirmed with my team that our Auth0 server does require an 
Audience
even from 
clientSecret
flow. So, I believe to make this work with the Auth server we do need Audience properly set.
y

Yee

10/05/2022, 6:48 PM
that is a little weird.
@Haytham Abuelfutuh will need help
i might be doing something wrong then… cuz it appears as if the golang std lib for client credentials doesn’t work with auth0.
a

Andrew Achkar

10/06/2022, 4:16 PM
@Yee I believe audience can be passed in if it is set in the tokenSource’s Config’s EndpointParams. See https://github.com/golang/oauth2/commit/01b79d94479495d23e14e5ddc8f85a9271aff260
s

Shahwar Saleem

10/06/2022, 4:18 PM
Thanks @Andrew Achkar for pointing that out, I created this issue: https://github.com/flyteorg/flyte/issues/2959 Also added above suggestion in it. @Yee @Haytham Abuelfutuh @Prafulla Mahindrakar
a

Andrew Achkar

10/06/2022, 5:21 PM
Might be that the only change needed is here? https://github.com/flyteorg/flyteidl/blob/1fb8cf18b290ff7c65b860f2f4423a1ae99e3d45/clients/go/admin/token_source_provider.go#L155-L175 Would then need to propagate flyteidl update to other components.
h

Haytham Abuelfutuh

10/06/2022, 5:26 PM
@Andrew Achkar so the problem is, as Yee pointed out, that the underlying golang library we use: https://cs.opensource.google/go/x/oauth2/+/b44042a4:clientcredentials/clientcredentials.go Doesn’t set audience on the request to issue tokens from IdP
the request sent to issue a token doesn’t include Audience (not even as an optional field)… it seems to me that Auth0's implementation doesn’t follow the standard
a

Andrew Achkar

10/06/2022, 5:29 PM
I think I follow, but the linked commit above shows how oauth2 library did add support for setting audience in the request via the EndpointParams.
It isn’t a named “audience” argument, but EndpointParams is flexible enough to support it.
h

Haytham Abuelfutuh

10/06/2022, 5:30 PM
aha… I missed that part… is that in a released version of oauth2?
a

Andrew Achkar

10/06/2022, 5:31 PM
it was merged Mar 13, 2017, so I hope so 🙂
h

Haytham Abuelfutuh

10/06/2022, 5:40 PM
If you want to modify our code to populate that, I’m all for it! Here is what I think needs to happen: 1. This PR needs to go in to create an audience field in Admin’s metadata response. 2. We need to follow up with a PR in admin to populate it here 3. Follow up PR in flyteidl to consume it here (see how scopes and tokenUrl is done)
y

Yee

10/06/2022, 5:52 PM
do we also want to add an audience field to the config object?
s

Shahwar Saleem

10/06/2022, 5:55 PM
Yes, config for Audience also needs to be provided from every service that uses ClientSecret. AFAIK these services are 
propeller
, 
scheduler
, and 
flytectl
.
y

Yee

10/06/2022, 5:58 PM
oh i think not… just saw the deprecated comment.
it’s needed yeah @Shahwar Saleem but it will discover it dynamically from admin.
s

Shahwar Saleem

10/12/2022, 6:40 PM
Thanks to Prafulla for implementing the changes required in 
flyteadmin
and 
flyteidl
for providing Audience. I was successfully able to create flytescheduler and propeller images. After I supply audience value, propeller is created fine. But scheduler is continuously in PodInitialization, unable to debug it further I have supplied the values to audience in my config maps like:
Copy code
admin.yaml: |
    admin:
      clientId: <clientId>
      clientSecretLocation: /etc/secrets/client_secret
      audience: <audience_value>
      endpoint: flyteadmin:81
      insecure: true
CC: @Yee please let me know what could I be doing wrong
y

Yee

10/12/2022, 6:42 PM
what’s the error message?
s

Shahwar Saleem

10/12/2022, 6:44 PM
All I see is that 
flytescheduler-check
is in CrashLoopBackOff state:
Copy code
terminated
Reason: Error - exit code: 2
Started at: 2022-10-12T14:39:43-04:00
Finished at: 2022-10-12T14:39:44-04:00
y

Yee

10/12/2022, 6:44 PM
can you get the logs?
i’m not caught up with the changes. what was made?
s

Shahwar Saleem

10/12/2022, 6:44 PM
Also, I checked admin logs and looks like Auth is working, as I see warnings related to the task only. So no 
Audience
error anymore.
y

Yee

10/12/2022, 6:44 PM
i’m a little confused.
are we still doing the original plan? use the client credentials flow with auth0?
did someone make the change to make use of the new audience field in the client credentials flow in the admin client?
s

Shahwar Saleem

10/12/2022, 6:46 PM
Yes sticking to the original plan. Prafulla made these PRs for me to test: https://github.com/flyteorg/flyteadmin/pull/485 https://github.com/flyteorg/flyteidl/pull/329
y

Yee

10/12/2022, 6:46 PM
i am not seeing that.
that pr doesn’t touch the client credentials flow
s

Shahwar Saleem

10/12/2022, 6:49 PM
I dont understand, what do you not see? I think this PR: https://github.com/flyteorg/flyteidl/pull/329/files does change 
token_source_provider
file to return 
EndPointParams
. Which are then going to be used to return Audience parameter.
y

Yee

10/12/2022, 6:49 PM
unf there’s still some work to do i think.
but how is that audience parameter passed to auth0
that still needs to be done.
s

Shahwar Saleem

10/12/2022, 6:51 PM
Copy code
audienceValue := cfg.Audience
		if len(audienceValue) == 0 {
			audienceValue = clientMetadata.Audience
		}

		tokenProvider, err = NewClientCredentialsTokenSourceProvider(ctx, cfg, scopes, tokenURL, audienceValue)
y

Yee

10/12/2022, 6:52 PM
oh this pr sorry i was looking at #314
s

Shahwar Saleem

10/12/2022, 6:52 PM
I think that is what above PR is doing, getting Audience from the config, converting it to an 
EndPointParams
to be passed.
Yes, I shared 2 new PR links which does what is said.
y

Yee

10/12/2022, 6:52 PM
and you pulled this pr into your propeller build?
and propeller is able to send eventing back to admin?
s

Shahwar Saleem

10/12/2022, 6:53 PM
Yes, I pulled these PRs to build new 
flyteadmin
flytepropeller
flytescheduler
y

Yee

10/12/2022, 6:53 PM
did you kick off a run? i dont’ think there’s traffic from propeller to admin until something is being run
s

Shahwar Saleem

10/12/2022, 6:55 PM
yes, I did. Workflow goes into unknown state...
y

Yee

10/12/2022, 6:56 PM
so auth is probably failing there too
s

Shahwar Saleem

10/12/2022, 6:57 PM
I see propeller has same
Copy code
No audience parameter was provided, and no default audience has been configured
y

Yee

10/12/2022, 6:57 PM
can you confirm first that the new client works? like just run the client as a unit test with golang and plug in the audience and secret/id
s

Shahwar Saleem

10/12/2022, 6:58 PM
I ll try that out and try to debug. Thanks!
y

Yee

10/12/2022, 7:00 PM
can you also confirm that the audience is being returned by the admin endpoint?
s

Shahwar Saleem

10/12/2022, 7:12 PM
How can I access this endpoint from k8s deployment?
y

Yee

10/12/2022, 7:20 PM
how do you access your flyte UI?
the flyte console
s

Shahwar Saleem

10/12/2022, 7:20 PM
yes
y

Yee

10/12/2022, 7:20 PM
yes it does?
s

Shahwar Saleem

10/12/2022, 7:20 PM
I have console URL that I access the UI through
Yup, that endpoint is returning a valid audience value configured by me.
y

Yee

10/12/2022, 7:22 PM
cool
then let’s confirm that a go unit test that hits auth0 also works
and then we can sort out what’s in between that’s breaking
s

Shahwar Saleem

10/20/2022, 5:41 PM
So updating this thread. Thanks to @Prafulla Mahindrakar again for coming through and adding PRs for new flyteidl and flyteadmin supporting audience in clients credential workflows. We were able to successfully run workflows after authentication on Flyte on Monday.
k

Ketan (kumare3)

10/20/2022, 8:37 PM
@Shahwar Saleem this is awesome news
s

Shahwar Saleem

10/20/2022, 9:43 PM
Thanks to team Flyte for their continuous support!
k

Ketan (kumare3)

10/21/2022, 3:02 AM
Thank you @Shahwar Saleem
Please help us spread awareness
j

Jonathan Lamiel

12/18/2022, 5:38 PM
Hello there, anyway to share how to setup auth0 with Flyte (maybe which version we need to use as well?) We'll need to do an equivalent setup soon
k

Ketan (kumare3)

12/18/2022, 5:43 PM
@Jonathan Lamiel it should work, but no docs
That are specific
j

Jonathan Lamiel

12/20/2022, 2:44 PM
Ok, I’m trying to setup it and got the same error on the propeller:
Copy code
No audience parameter was provided, and no default audience has been configured
and this on the admin side:
Copy code
{"json":{"src":"cookie.go:88"},"level":"debug","msg":"Existing [flyte_idt] cookie found","ts":"2022-12-20T14:30:01Z"}
{"json":{"src":"cookie.go:88"},"level":"debug","msg":"Existing [flyte_at] cookie found","ts":"2022-12-20T14:30:01Z"}
{"json":{"src":"cookie.go:88"},"level":"debug","msg":"Existing [flyte_rt] cookie found","ts":"2022-12-20T14:30:01Z"}
{"json":{"src":"cookie.go:88"},"level":"debug","msg":"Existing [flyte_user_info] cookie found","ts":"2022-12-20T14:30:01Z"}
{"json":{"src":"handlers.go:235"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-12-20T14:30:01Z"}
{"json":{"src":"token.go:83"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-12-20T14:30:01Z"}
{"json":{"src":"handlers.go:245"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-12-20T14:30:01Z"}
I’m using the 
v1.2.1
images and added the audience conf as below
Copy code
thirdPartyConfig:
          flyteClient:
            clientId: clientId
            redirectUri: <https://xxxx.auth0.com/callback>
            scopes:
            - offline
            - all
            audience: <https://xxxx.auth0.com/api/v2/>
And the audience is correctly return by is correctly return by the admin endpoint 
/config/v1/flyte_client
I’m not sure what I’m doing wrong there
k

Ketan (kumare3)

12/20/2022, 3:32 PM
Cc @Prafulla Mahindrakar do you know off hand
p

Prafulla Mahindrakar

12/20/2022, 4:26 PM
We haven’t merged this PR https://github.com/flyteorg/flyteidl/pull/329/files and isn’t part of the release yet. Let me prioritize the reviews on it and get it checkedin and will require followup PR’s in flytepropeller to consume it. @Jonathan Lamiel if you want try it out yourself then you can use the idl changes directly in your propeller and build it which i can help you with it or else i will give you released version once the code is checkedin
j

Jonathan Lamiel

12/20/2022, 4:34 PM
Hooo I see apologized I only looked at the first PR 😅 and didn’t realized it wasn’t merged. Yeah, I’ll try to build it on my side to test it
p

Prafulla Mahindrakar

12/20/2022, 5:44 PM
np. Let me know if you run into issues. Meanwhile will try to get this checkedin
j

Jonathan Lamiel

12/23/2022, 3:05 PM
Hey there, so I’m trying to build a release for flyteidl but got issues such as:
Copy code
Error: unknown command "RedisClient" for "mockery"
Error: unknown command "HandlerFactory" for "mockery"
I’m not a go dev, so it surely is an issue on my setup 😅
k

Ketan (kumare3)

12/23/2022, 4:28 PM
Seems like it - mocks
p

Prafulla Mahindrakar

12/23/2022, 8:11 PM
Hi @Jonathan Lamiel, you dont need to create a release and you can directly consume this by building a new propeller image with the changes. In flytepropeller repo, you can run • 
go get <http://github.com/flyteorg/flyteidl|github.com/flyteorg/flyteidl>
<http://github.com/flyteorg/flyteidl@956c7a259b50a6607aa82a9d3cc0bcbe51919f84|github.com/flyteorg/flyteidl@956c7a259b50a6607aa82a9d3cc0bcbe51919f84>
which uses the commit from the PR • 
go mod tidy
• 
make docker_build
. this will build the propeller image with the idl changes • Push the built image to your local image repo and use the same in your flyte deployment. With the change you can now configure flytepropeller to send the audience field when having flyteadmin relay auth with auth0 provider. In your propeller config map you should be able add the audience key and value https://github.com/flyteorg/flyteidl/pull/329/files#diff-91f1e2cdbc64e0a780abe3c3eddfeb4bc61a1e099e7393a1331ff7f27be30c5cR56 in the admin section
j

Jonathan Lamiel

01/03/2023, 2:41 PM
Hey @Prafulla Mahindrakar Happy new year! I finally been able to test it out. It works like a charm now using the Console or API with a provided bearer 🙂 As we want to setup a CI and via the Python SDK, I think we would need to update this too correct? I’ll update the thread once all good
p

Prafulla Mahindrakar

01/03/2023, 5:47 PM
good to hear that it worked for you . yes to use flytekit a corresponding change would be needed there too. Would you be willing to make that contribution to the sdk which i suppose shouldn’t be much and also since you have an environment where you have auth setup with a non-default audience then it would be easily testable too. Let us know what you think
j

Jonathan Lamiel

01/03/2023, 7:32 PM
With pleasure, just make it works 😉 so I’ll try to create a PR by the end of the week. But it will require your PRs to be merge before no for flyteidl? (I built it locally for now)
p

Prafulla Mahindrakar

01/03/2023, 9:03 PM
Awesome. yeah. will try to get those flyteidl ones by then . Thanks Jonathan.
k

Ketan (kumare3)

01/04/2023, 2:50 AM
Can I know more about this. I am working on making the sdk auth layer much better
p

Prafulla Mahindrakar

01/04/2023, 3:46 AM
@Ketan (kumare3) in the above case, the auth server is configured with no default audience and the client needs to have this audience configuration when requesting a token from the auth server and using that in the flyteadmin resource server for requesting resources. flyteidl changes have this changes and needs to go in along with flytekit changes to use this config.
23 Views
Powered by