https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
k

karthikraj

10/17/2022, 10:30 PM
Hi , I am trying to integrate okta with Flyte. Following this page for setting up appAuth and userAuth. First tried only setting up userAuth in configmap. It worked, okta is enabled. Then included appAuth to configmap and re-deployed, facing the below error in flytescheduler-check container of flytescheduler. Using the same client_id for userAuth and appAuth.
Copy code
panic: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 400 Bad Request
Response: {"errorCode":"invalid_client","errorSummary":"Invalid value for 'client_id' parameter.","errorLink":"invalid_client","errorId":"oae4BTFncguRFCNZUfFEpIFhA","errorCauses":[]}

goroutine 1 [running]:
main.main()
        /go/src/github.com/flyteorg/flyteadmin/cmd/scheduler/main.go:12 +0x85
This is my sample configmap for this.
Copy code
configmap:
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          baseUrl:  <https://www.oktadev.com/oauth2/default>
        thirdPartyConfig:
          flyteClient:
            clientId:  "a1b2c3xxxxyz9N3" 
            redirectUri: <https://flytedev.myapp.com/callback>
            scopes:
            - offline
            - all
      userAuth:
        openId:
          baseUrl: <https://www.oktadev.com/oauth2/default>
          scopes:
            - profile
            - openid
          clientId: "a1b2c3xxxxyz9N3"
      authorizedUris:
        - <https://flytedev.myapp.com>
        - <http://flyteadmin:80>
        - <http://flyteadmin.flyte.svc.cluster.local:80>
t

Tyler Su

10/17/2022, 11:16 PM
Hi there karthikraj, we’ve seen your question and we’ll get you some assistance with this error soon! Please hang tight!
k

karthikraj

10/18/2022, 1:31 AM
Thank you 🙂
s

Samhita Alla

10/18/2022, 4:35 AM
cc: @Prafulla Mahindrakar
p

Prafulla Mahindrakar

10/18/2022, 5:52 AM
Hi @karthikraj, did you follow these steps https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#oauth2-authorization-server
Copy code
You should have three integrations total - one for the web interface, one for Flytectl, and one for Flytepropeller.
Flytepropeller shares the secrets currently with flytescheduler. (and cluster-resource manager) Configuration under appAuth controls propeller,scheduler , flytectl thirdPartConfig controls specifically flytectl config. This is the section which loads the flyte-secret-auth which gets mounted on both scheduler and propeller for doing auth with flyteadmin pod https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values.yaml#L290-L297 I think you have misconfigured clientId here or you have not created this clientId in your auth provider. This paricular clientId needs to use clientsecret auth flow.
k

karthikraj

10/18/2022, 11:03 PM
Thank you sm for this. Let me check from myside and get back. @Prafulla Mahindrakar
@Prafulla Mahindrakar So, As per the documentation, there is going to be two Client ids. One for Flytectl which will have only ClientId and no secret -> goes under appAuth Another will have Client Id and secret -> goes under userAuth I think i am using same clientid in both the places. Let me check that. This is new error from flytescheduler pod. Is this because of using same clientId in both places?
Copy code
panic: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"unauthorized_client","error_description":"The client is not authorized to use the provided grant type. Configured grant types: [authorization_code]."}

goroutine 1 [running]:
main.main()
        /go/src/github.com/flyteorg/flyteadmin/cmd/scheduler/main.go:12 +0x85
p

Prafulla Mahindrakar

10/19/2022, 3:36 AM
What is the clientId you see in scheduler. You can check this using 
kubectl get configmap -n flyte flytescheduler -o yaml
and check the admin section . This should be third one shared amongst propeller and scheduler. You need 3 clientId’s. Two of them are configured in the admin config and the third one is used directly in client configs of the respective apps (propeller and scheduler)
and you would need to have authorization_code grant provided for your auth client that you use.
k

karthikraj

10/19/2022, 3:44 AM
oh okay.. let me check. and you would need to have authorization_code grant provided for your auth client that you use. --->oh, This should be something to be provided in the okta app? (okta is managed by separate team, so I am not aware of this, so checking with you.)
p

Prafulla Mahindrakar

10/19/2022, 4:13 AM
yes this grant has to be done through okta.
k

karthikraj

10/27/2022, 12:42 AM
Hi @Prafulla Mahindrakar, I was unavailable for past few days and I am continued to working on the above. I am working with my okta team and here i have few clarification as I could see same Client Ids are configured in apps(scheduler&propeller) and admin I am sorry for asking it again as I am not really clear when okta team came up with the few questions😞 By following the points mentioned here -> https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#oauth2-authorization-server Points 1-6: This is for creating Base url by passing in flyte url. Example for base url : https://mydev.okta.com/oauth2/default Now we got base url. Points 7-10: This is for flytectl integration. We will add redirect_url and finally we will get the ClientId and no secret. (This is our FIRST CLIENT ID). Points 11-14: This is for propeller(shared with scheduler too). Here we will get ClientID & Client secret. (This is our SECOND CLIENT ID)
Copy code
auth:
    appAuth:
        authServerType: External
        externalAuthServer:
            baseUrl: <BASE_URL FROM POINT 1-6>                              #####POINT 1-6
        thirdPartyConfig:
            flyteClient:
                clientId: <CLIENT_ID FROM POINT 7-10>                       #####POINT 7-10
                redirectUri: <REDIRECT_URI FROM POINT 7-10>                 #####POINT 7-10
                scopes:
                - offline
                - all
    userAuth:
        openId:
            baseUrl: <BASE_URL FROM POINT 1-6>                              #####POINT 1-6
            scopes:
            - profile
            - openid
            clientId: <CLIENT_ID FROM POINT 11-14>                       #####POINT 11-14
• I have verified that flyte-secret-auth is created with the credentials from point 11-14 and it is shared between sched&propeller by setting the values in adminOauthClientCredentials (here) Please tell me if my understanding is correct? As per your message, Which are three client ids as I could see only two from the points?
p

Prafulla Mahindrakar

10/27/2022, 6:44 AM
I see the confusion . you are actually using the flyteconsole (webinterface) clientId for propeller/scheduler too which is not correct. • web interface (flyteconsole) -> userAuth.openId.clientId whose secret is set using this section https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#apply-configuration • flytectl (cli tool) -> appAuth.thirdPartyConfig.clientId (no secret as it uses Pkce flow) • For propeller and scheduler -> this clientId is not maintained in above appAuth config but instead directly in the clients throught flyte-secret-auth created using https://github.com/flyteorg/flyte/blob/815abde798c09bd1ac537497214323a0078fa008/charts/flyte-core/values.yaml#L290-L297 the scopes are shared by flytectl, flytescheduler, flyepropller through appAuth.thirdPartyConfig.flyteClient.scopes but can be overriden here https://github.com/flyteorg/flyte/blob/815abde798c09bd1ac537497214323a0078fa008/charts/flyte-core/values.yaml#L558-L562 corresponding config here https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/config.go#L55
k

karthikraj

10/31/2022, 1:53 AM
Thank you sm for taking your time and explaining. This is clear now 😊 I am talking with my okta team. Will get back if any there are any issues. @Prafulla Mahindrakar
@Prafulla Mahindrakar I have communicated the info. My okta managing team is asking to schedule a call to understand this better. Could some one from your team is possible to join to sort this out? It will be scheduled based on the availability from both the teams.
Hi Team/@Prafulla Mahindrakar, Will it be possible from your side to help us with a short call?
p

Prafulla Mahindrakar

11/02/2022, 12:45 PM
Can we set up some time for tomorrow. I work in IST zone.
If your team is available anytime within next 3 hours then we can do it today.
k

karthikraj

11/02/2022, 12:48 PM
okay.. Let me check with our okta team....
@Prafulla Mahindrakar Unfortunately, yesterday they were not available. Can we have call today(Nov 3, 7.45PM to 8.15PM IST)? I Will schedule a meeting. works for you?
@Prafulla Mahindrakar What would be the best time for you?
p

Prafulla Mahindrakar

11/04/2022, 2:41 AM
Yes sure works for me .(pmahindrakar@union.ai)
k

karthikraj

11/04/2022, 1:47 PM
@Prafulla Mahindrakar Hi, Did you get an invite from me?
p

Prafulla Mahindrakar

11/04/2022, 1:48 PM
Yes I did
k

karthikraj

11/04/2022, 2:05 PM
okay.. Thank you 🙂
Thank you so much @Prafulla Mahindrakar That was very helpful 🙂
p

Prafulla Mahindrakar

11/04/2022, 2:54 PM
Let me know if you face any issues.
k

karthikraj

11/04/2022, 3:21 PM
Sure...👍
Hi @Prafulla Mahindrakar I have got clientIds and had setup them for Flyteconsole, Flytepropeller, Flytectl . Flytescheduler pod is still having a problem. I am getting below error in flytescheduler-check container. Do we need to have this grant type [authorization_code] assigned to Propeller/scheduler clientid? I dont see the list of grant types required for propeller clientId anywhere in the documentation. What are the other grant types required here?
Copy code
panic: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"unauthorized_client","error_description":"The client is not authorized to use the provided grant type. Configured grant types: [authorization_code]."}
p

Prafulla Mahindrakar

11/21/2022, 6:05 PM
Hi @karthikraj, this flytepropeller which is shared with scheduler should be using the same grant types as ones for flyteconsole. i.e client credentials and authorization code . Attaching an image from okta config Yeah i think these are not documented currently but would be great if you can help file a doc issue for it and also if possible help with a PR on it once you have things working .
k

karthikraj

11/21/2022, 8:35 PM
@Prafulla Mahindrakar Now, I am getting this error. What could be the reason? I have set the credentials as per the doc.
Copy code
panic: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken
p

Prafulla Mahindrakar

11/22/2022, 5:21 AM
Can you check what error do you see in the admin logs which would provide you more details
We can probably get on a call to debug this if that’s easier
k

karthikraj

11/22/2022, 3:01 PM
I dont see any issues with admin pods. Here is the log from it. Sure, call would be really helpful. What time works for you? @Prafulla Mahindrakar
Copy code
]$ kubectl logs flyteadmin-f884d7f97-b99mb -n flyte
time="2022-11-20T23:58:22Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/clusters.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
{"json":{},"level":"warning","msg":"stow configuration section missing, defaulting to legacy s3/minio connection config","ts":"2022-11-20T23:58:23Z"}
p

Prafulla Mahindrakar

11/22/2022, 3:03 PM
Can we get on a call now if you are available.
k

karthikraj

11/22/2022, 3:06 PM
Sure, let me ping you the zoom link
@Prafulla Mahindrakar Thanks for joining the call and helping me to resolve the issue. We have added allowedAudience in the AuthServer which resolved the issue.
Since pods are up. I am testing by registering the flyte workflows with this new auth setup
@Prafulla Mahindrakar Thank for helping me out in the personal chat. The below config works in my local
Copy code
admin:
  endpoint: dns:///flyte.dev.xxx.xxx.com
  authType: ClientSecret
  clientId: <id>
  clientSecretLocation: /home/kkanagar/.flyte/client_secret
  insecure: false
  scopes: [ "all" ]
The workflow is registered in the Flyte and able to see the workflow in the console but the status is showing UNKNOWN.
This is the log from admin pod.
Copy code
{"json":{"exec_id":"fb21c31a2c48148b4b51","src":"execution_manager.go:381"},"level":"warning","msg":"Failed to fetch override values when assigning task resource default values for [resource_type:WORKFLOW project:\"examples\" domain:\"hbomax\" name:\"flyte.workflows.hi_world.my_wf\" version:\"3PMLBzY2tBl5bHSxXauKHQ==\" ]: Resource [{Project:examples Domain:hbomax Workflow:flyte.workflows.hi_world.my_wf LaunchPlan: ResourceType:TASK_RESOURCE}] not found","ts":"2022-11-22T21:27:44Z"}
{"json":{"exec_id":"fb21c31a2c48148b4b51","src":"execution_manager.go:385"},"level":"debug","msg":"Assigning task requested resources for [resource_type:WORKFLOW project:\"examples\" domain:\"hbomax\" name:\"flyte.workflows.hi_world.my_wf\" version:\"3PMLBzY2tBl5bHSxXauKHQ==\" ]","ts":"2022-11-22T21:27:44Z"}
{"json":{"src":"queues.go:43"},"level":"debug","msg":"refreshing execution queues","ts":"2022-11-22T21:27:44Z"}
{"json":{"exec_id":"fb21c31a2c48148b4b51","src":"queues.go:73"},"level":"warning","msg":"Failed to fetch override values when assigning execution queue for [{ResourceType:WORKFLOW Project:examples Domain:hbomax Name:flyte.workflows.hi_world.my_wf Version:3PMLBzY2tBl5bHSxXauKHQ== XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] with err: Resource [{Project:examples Domain:hbomax Workflow:flyte.workflows.hi_world.my_wf LaunchPlan: ResourceType:EXECUTION_QUEUE}] not found","ts":"2022-11-22T21:27:44Z"}
{"json":{"exec_id":"fb21c31a2c48148b4b51","src":"queues.go:109"},"level":"info","msg":"found no matching queue for [{ResourceType:WORKFLOW Project:examples Domain:hbomax Name:flyte.workflows.hi_world.my_wf Version:3PMLBzY2tBl5bHSxXauKHQ== XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]","ts":"2022-11-22T21:27:44Z"}
{"json":{"exec_id":"fb21c31a2c48148b4b51","src":"execution_manager.go:529"},"level":"info","msg":"getting the workflow execution config from application configuration","ts":"2022-11-22T21:27:44Z"}
{"json":{"src":"handlers.go:209"},"level":"debug","msg":"Found existing metadata Bearer eyJraWQiOiJPcTYzbzFRWUlDaDJVTExmTFM4bDRyRlRRbFQzcTRxNGljcW0xZVVDS1JFIiwiYWxnIjoiUlMyNTYifQ.eyJ2ZXIiOjEsImp0aSI6IkFULi1jdUwwMGl4eWRLNVFNZzNqSWJwbGZOakdRd2g0aGJtUTF1bzVEY1RaLWMiLCJpc3MiOiJodHRwczovL3R3ZGV2Lm9rdGFwcmV2aWV3LmNvbS9vYXV0aDIvZGVmYXVsdCIsImF1ZCI6ImFwaTovL2RlZmF1bHQiLCJpYXQiOjE2NjkxNTI0NjMsImV4cCI6MTY2OTE1NjA2MywiY2lkIjoiMG9hMWY0djhvZGs5OE5peFUwaDgiLCJzY3AiOlsiYWxsIl0sInN1YiI6IjBvYTFmNHY4b2RrOThOaXhVMGg4In0.Lw6aEcITlfkkul3zTbkeM0N27s3dmIA8JcDrheQ1iruBNsIVLzoIB_L7ezeSlzK0yDlh43RUbX1gGEQc_ELOuveeCv1iU-GOR5UoiaacdF4t5c7aOm5SGX5Hchmdh-MIe4dAsiUOIN1nABNrN8E44I9FsEB0HxWd-JsQx7W7-vs_KSf8aB0orooupN9xFEOlPImMyjNZpdFBzbTmRAVgxpmbaPk4-4nUy4CjejVFi1jUsD_t4Q-84fuhctzm0XdSUUnPGgUnFOId95nH_N3VGLRyhQd_a9Z2nskwLCA-_Er_gupQJfksFEC-kZbr9Yql0Z4ITauL5zE0OpKiH18RHQ","ts":"2022-11-22T21:27:44Z"}
{"json":{"src":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-11-22T21:27:44Z"}
{"json":{"src":"handlers.go:193"},"level":"debug","msg":"gRPC server info in logging interceptor [0oa1f4v8odk98NixU0h8]method [/flyteidl.service.AdminService/GetExecution]\n","ts":"2022-11-22T21:27:44Z"}
{"json":{"src":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"handlers.go:247"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"token.go:104"},"level":"debug","msg":"Could not retrieve id token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"handlers.go:247"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"token.go:104"},"level":"debug","msg":"Could not retrieve id token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-11-22T21:27:59Z"}
{"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-11-22T21:27:59Z"}
p

Prafulla Mahindrakar

11/23/2022, 6:16 AM
Can you check the propeller logs, seems its not able to report the progress when talking to admin and hence giving you those auth errors. Check if its due to same scope 
all
not provided in the propeller config map. If you have configured the scopes in the admin configmap then we don’t have to explicity set it .
Copy code
thirdPartyConfig:
            flyteClient:
              clientId: <flytectl-client-id>
              redirectUri: <http://localhost:53593/callback>
              scopes:
              - all
This section mentions that scope in the thirdPartyConfig https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#id1 Can you check if you have configured it in your flyteadmin-base config
k

karthikraj

11/23/2022, 5:03 PM
@Prafulla Mahindrakar I have ensured that the scopes are set fine. This is the below error keep coming from propeller also similar error from admin pod as well.
Copy code
E1123 17:01:25.016586       1 workers.go:102] error syncing 'examples-hbomax/f91bb33abf49a4eada70': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]
E1123 17:01:25.017858       1 workers.go:102] error syncing 'examples-hbomax/f8ef2c9513b334b17b45': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]
• pods are up • Able to register the workflow to flyte But the workflow is not moving to running state. it is still in unknown.. if we check admin & propeller logs, seeing the above error.
@Prafulla Mahindrakar Thank you so much for your time today over a call to fix the issue faced. We are able to register and successfully run the workflow after redeploying the flytepropeller.
e

Evan Sadler

01/04/2023, 5:42 PM
@karthikraj we are experiencing a similar issue. What did you have to change?
13 Views
Powered by