https://flyte.org logo
#flyte-deployment
Title
# flyte-deployment
s

Shahwar Saleem

08/02/2022, 4:59 PM
Hi Flyte Fellows, Has anyone used Auth0 or Okta with Flytectl before? I followed the Okta instructions via OIDC and was able to verify Auth works with Auth0 and Okta for Flyte Console. But after following instructions for flytectl I am still getting following error:
Copy code
flytectl get project 
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"
I was wondering if anyone had any insights what could possibly cause this? The Auth server is configured properly and the error is coming from Flyte Admin, which means it does not trust the flytectl issuing commands.
s

Samhita Alla

08/03/2022, 5:01 AM
cc: @Prafulla Mahindrakar
p

Prafulla Mahindrakar

08/03/2022, 5:51 AM
I think you have self signed or cert signed by unknow authority. You can choose to skip the verification of the cert using
insecureSkipVerify
flag https://github.com/flyteorg/flyteidl/blob/7b1fd01ff429d28755765204f00dea57598d8851/clients/go/admin/config.go#L41 But its not recommended for prod usecases and should be used only for testing.
This you would need to update in the config.yaml for flytectl which is usually in ~/.flyte/ directory.
you can also pass in thi from flytectl commanline flag with
Copy code
--admin.insecureSkipVerify  true
more flags are available through help https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl.html
s

Shahwar Saleem

08/03/2022, 2:44 PM
Yes, I was aware of this flag. But since we are evaluating Flyte for prod usecases, I would like to make sure what is not right when it comes to the certificate and sign an appropriate one for flytectl and flyte console both.
p

Prafulla Mahindrakar

08/03/2022, 4:24 PM
How did you provision your certificate and which certifying authority is being used to sign it.
s

Shahwar Saleem

08/04/2022, 5:21 PM
Hi @Prafulla Mahindrakar I think I changed my cert config and now above error is gone. I am now getting a different error:
Copy code
root@0d4683ee86d5:/# flytectl get project
Error: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken
I am also wondering how
Pkce
is used within Flyte to talk to FlyteCTL? I talked to a person who handles Auth within my company, they have never seen Pkce on a Command Line. Could you please advise on this?
p

Prafulla Mahindrakar

08/05/2022, 5:57 AM
Hi @Shahwar Saleem by default flytectl uses pkce which doesn’t require to store shared client secrets and it requires browser access to complete the auth flow. It will connect you to configured OIDC provider and complete the flow. It seems you might be on a machine which doesn’t have browser access, If thats the case then you can change the authType from pkce to client secret eg : config.yaml or you can pass these flags from cli too
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyte.org
  # Change insecure flag to ensure that you use the right setting for your environment
  insecure: false
  clientId: *********
  authType: ClientSecret
  clientSecretLocation: /home/runner/secret_location
logger:
  # Logger settings to control logger output. Useful to debug logger:
  show-source: true
  level: 1
s

Shahwar Saleem

08/31/2022, 5:52 PM
Auth Update on this thread. @Prafulla Mahindrakar Did implement Device Authentication Flow for flytectl under the same PR. Here are some details after that: • I was able to authenticate using Device Flow on the browser by making following changes: ◦ Configured Flyte Admin Config Map according to this ◦ Needed to add
metadataUrl
to be
.well-known/openid-configuration
. This is a suffix to baseURL internally. ◦ Needed to add new clientId which was provided by Auth Server App. ◦ Needed to change the flyteadmin image to a new one corresponding to DeviceFlow changes:
<http://ghcr.io/flyteorg/flyteadmin:v1.1.37-deviceauth|ghcr.io/flyteorg/flyteadmin:v1.1.37-deviceauth>
. This change needs to happen in flyteadmin deployment accross all occurrences of the flyteadmin image. ◦ Needed to change FlyteCTL config to something like this:
Copy code
admin:
  endpoint: <URL>
  authType: DeviceFlow
  insecure: false
  caCertFilePath: <certfilepath>
  deviceFlowConfig:
    deviceFlowTimeout: 200s
logger:
  show-source: true
  level: 20
After authenticating, I am getting an error which does signify there is an issue with JWT response. Apparently, for our usecase an
audience
needs to be specified while requesting auth from Auth0 server. Currently,
flyteadmin
is not using audience field and due to missing Audience, Auth Server is not able to complete the Device Authentication flow, because the token after authorization is a default one, which Auth server is not accepting. A few things: • We would need full Auth support including Audience field to fully utilize Authentication with Flytectl. • The team is now outside POC and implementing a full Flyte Deployment which will need full Authentication capability for Flyteconsole as well as FlyteCTL. CC: @Andrew Achkar @Geoff Salmon @Haytham Abuelfutuh
I also would like to add that we really appreciate the quick response and PR from Team Flyte and @Prafulla Mahindrakar has been working with us on this. Much appreciated.
h

Haytham Abuelfutuh

08/31/2022, 9:32 PM
This is great to hear @Shahwar Saleem thank you for the update! The goal is to make this very intuitive to use/configure. I would really appreciate it if you can help us with the Auth0 config part (to add another section under the auth doc for Auth0)… I would like to understand a big deeper the missing audience problem. Is this something that can be addressed in the request for a token either on the flytectl side or admin? is it a configuration issue/change in Auth0? Also why is there a need for setting caCertFilePath? is it because your host doesn’t have a publicly accessible/verifiable SSL Cert?
p

Prafulla Mahindrakar

09/01/2022, 7:04 AM
So it seems auth0 is always sending opaque token on device flow path though admin flow require a JWT and parser fails here in admin https://github.com/flyteorg/flyteadmin/blob/master/auth/authzserver/resource_server.go#L33 From Auth0 community pages it seems its recommended to pass in the audience param when requesting an access token inorder to receive a JWT https://community.auth0.com/t/why-is-it-necessary-to-pass-the-audience-parameter-to-receive-a-jwt/11412/6 But the doc also do mention configuring a default audience parameter on the auth0 auth server. This is how i tested my changes using okta which worked. Wanted to check why can’t auth team make this change . If we set it at the auth server side , the token issued will be verified by admin using the allowedAudience field and will accept or reject the token based on if the token is intended for it or not. Now coming to another point that clientCredentials flow works with same auth server and flyteconsole is able to pull resources from flyteadmin resource server. And browsing through the golang aouth library and tracing our code path, I couldn’t find that we are passing the aud info in that flow. So wondering how come clientCreds work then. The config used for clientCreds use this and this doesn’t pass audience field https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/token_source_provider.go#L156-L161 Oauth library expects any additional params eg audience to be passed in
Copy code
// EndpointParams specifies additional parameters for requests to the token endpoint.
	EndpointParams url.Values
https://github.com/golang/oauth2/blob/master/clientcredentials/clientcredentials.go#L44 which gets used here https://github.com/golang/oauth2/blob/master/clientcredentials/clientcredentials.go#L97 Sample test with aud https://github.com/golang/oauth2/blob/master/clientcredentials/clientcredentials_test.go#L25 But none of this config is passed from flyteadmin which is why i am uncertain that adding the aud field is going to solve the device flow issue. Wanted to test this with a custom build and verify before making a configurable change. Let me know what you all think .
s

Shahwar Saleem

09/02/2022, 3:03 PM
@Prafulla Mahindrakar Please let me know if there is any update on this or how are the timelines looking for Audience.
p

Prafulla Mahindrakar

09/02/2022, 6:38 PM
@Shahwar Saleem did we get a chance to check if adding the aud on auth server works. You can use the following draft pr in flyteidl for passing the audience from cli https://github.com/flyteorg/flyteidl/pull/314 . • Create new flytectl binary using it
Copy code
go mod edit -replace <http://github.com/flyteorg/flyteidl=github.com/flyteorg/flyteidl@4b475700a8df0e7736fed995ee84a4cc22e453fd|github.com/flyteorg/flyteidl=github.com/flyteorg/flyteidl@4b475700a8df0e7736fed995ee84a4cc22e453fd>
 go mod tidy
And then run using the audience by passing the right aud field for your app
Copy code
flytectl get projects --logger.level=6  --admin.authType DeviceFlow  --admin.deviceFlowConfig.audience <aud>
Ref used on how to pass aud https://auth0.com/docs/secure/tokens/access-tokens/get-access-tokens
121 Views