https://flyte.org logo
Join Slack
Powered by
Hi , I am trying to integrate okta with Flyte. Fo...
# flyte-support
d
Hi , I am trying to integrate okta with Flyte. Following this page for setting up appAuth and userAuth. First tried only setting up userAuth in configmap. It worked, okta is enabled. Then included appAuth to configmap and re-deployed, facing the below error in flytescheduler-check container of flytescheduler. Using the same client_id for userAuth and appAuth.
Copy code
panic: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 400 Bad Request
Response: {"errorCode":"invalid_client","errorSummary":"Invalid value for 'client_id' parameter.","errorLink":"invalid_client","errorId":"oae4BTFncguRFCNZUfFEpIFhA","errorCauses":[]}

goroutine 1 [running]:
main.main()
        /go/src/github.com/flyteorg/flyteadmin/cmd/scheduler/main.go:12 +0x85
This is my sample configmap for this.
Copy code
configmap:
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          baseUrl:  <https://www.oktadev.com/oauth2/default>
        thirdPartyConfig:
          flyteClient:
            clientId:  "a1b2c3xxxxyz9N3" 
            redirectUri: <https://flytedev.myapp.com/callback>
            scopes:
            - offline
            - all
      userAuth:
        openId:
          baseUrl: <https://www.oktadev.com/oauth2/default>
          scopes:
            - profile
            - openid
          clientId: "a1b2c3xxxxyz9N3"
      authorizedUris:
        - <https://flytedev.myapp.com>
        - <http://flyteadmin:80>
        - <http://flyteadmin.flyte.svc.cluster.local:80>
w
Hi there karthikraj, we’ve seen your question and we’ll get you some assistance with this error soon! Please hang tight!
d
Thank you 🙂
t
cc: @icy-agent-73298
i
Hi @delightful-lion-57360, did you follow these steps https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#oauth2-authorization-server
Copy code
You should have three integrations total - one for the web interface, one for Flytectl, and one for Flytepropeller.
Flytepropeller shares the secrets currently with flytescheduler. (and cluster-resource manager) Configuration under appAuth controls propeller,scheduler , flytectl thirdPartConfig controls specifically flytectl config. This is the section which loads the flyte-secret-auth which gets mounted on both scheduler and propeller for doing auth with flyteadmin pod https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values.yaml#L290-L297 I think you have misconfigured clientId here or you have not created this clientId in your auth provider. This paricular clientId needs to use clientsecret auth flow.
d
Thank you sm for this. Let me check from myside and get back. @icy-agent-73298
@icy-agent-73298 So, As per the documentation, there is going to be two Client ids. One for Flytectl which will have only ClientId and no secret -> goes under appAuth Another will have Client Id and secret -> goes under userAuth I think i am using same clientid in both the places. Let me check that. This is new error from flytescheduler pod. Is this because of using same clientId in both places?
Copy code
panic: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"unauthorized_client","error_description":"The client is not authorized to use the provided grant type. Configured grant types: [authorization_code]."}

goroutine 1 [running]:
main.main()
        /go/src/github.com/flyteorg/flyteadmin/cmd/scheduler/main.go:12 +0x85
i
What is the clientId you see in scheduler. You can check this using 
kubectl get configmap -n flyte flytescheduler -o yaml
and check the admin section . This should be third one shared amongst propeller and scheduler. You need 3 clientId’s. Two of them are configured in the admin config and the third one is used directly in client configs of the respective apps (propeller and scheduler)
and you would need to have authorization_code grant provided for your auth client that you use.
d
oh okay.. let me check. and you would need to have authorization_code grant provided for your auth client that you use. --->oh, This should be something to be provided in the okta app? (okta is managed by separate team, so I am not aware of this, so checking with you.)
i
yes this grant has to be done through okta.
d
Hi @icy-agent-73298, I was unavailable for past few days and I am continued to working on the above. I am working with my okta team and here i have few clarification as I could see same Client Ids are configured in apps(scheduler&propeller) and admin I am sorry for asking it again as I am not really clear when okta team came up with the few questions😞 By following the points mentioned here -> https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#oauth2-authorization-server Points 1-6: This is for creating Base url by passing in flyte url. Example for base url : https://mydev.okta.com/oauth2/default Now we got base url. Points 7-10: This is for flytectl integration. We will add redirect_url and finally we will get the ClientId and no secret. (This is our FIRST CLIENT ID). Points 11-14: This is for propeller(shared with scheduler too). Here we will get ClientID & Client secret. (This is our SECOND CLIENT ID)
Copy code
auth:
    appAuth:
        authServerType: External
        externalAuthServer:
            baseUrl: <BASE_URL FROM POINT 1-6>                              #####POINT 1-6
        thirdPartyConfig:
            flyteClient:
                clientId: <CLIENT_ID FROM POINT 7-10>                       #####POINT 7-10
                redirectUri: <REDIRECT_URI FROM POINT 7-10>                 #####POINT 7-10
                scopes:
                - offline
                - all
    userAuth:
        openId:
            baseUrl: <BASE_URL FROM POINT 1-6>                              #####POINT 1-6
            scopes:
            - profile
            - openid
            clientId: <CLIENT_ID FROM POINT 11-14>                       #####POINT 11-14
• I have verified that flyte-secret-auth is created with the credentials from point 11-14 and it is shared between sched&propeller by setting the values in adminOauthClientCredentials (here) Please tell me if my understanding is correct? As per your message, Which are three client ids as I could see only two from the points?
i
I see the confusion . you are actually using the flyteconsole (webinterface) clientId for propeller/scheduler too which is not correct. • web interface (flyteconsole) -> userAuth.openId.clientId whose secret is set using this section https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#apply-configuration • flytectl (cli tool) -> appAuth.thirdPartyConfig.clientId (no secret as it uses Pkce flow) • For propeller and scheduler -> this clientId is not maintained in above appAuth config but instead directly in the clients throught flyte-secret-auth created using https://github.com/flyteorg/flyte/blob/815abde798c09bd1ac537497214323a0078fa008/charts/flyte-core/values.yaml#L290-L297 the scopes are shared by flytectl, flytescheduler, flyepropller through appAuth.thirdPartyConfig.flyteClient.scopes but can be overriden here https://github.com/flyteorg/flyte/blob/815abde798c09bd1ac537497214323a0078fa008/charts/flyte-core/values.yaml#L558-L562 corresponding config here https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/config.go#L55
d
Thank you sm for taking your time and explaining. This is clear now 😊 I am talking with my okta team. Will get back if any there are any issues. @icy-agent-73298
👍 1
@icy-agent-73298 I have communicated the info. My okta managing team is asking to schedule a call to understand this better. Could some one from your team is possible to join to sort this out? It will be scheduled based on the availability from both the teams.
Hi Team/@icy-agent-73298, Will it be possible from your side to help us with a short call?
i
Can we set up some time for tomorrow. I work in IST zone.
If your team is available anytime within next 3 hours then we can do it today.
d
okay.. Let me check with our okta team....
👍 1
@icy-agent-73298 Unfortunately, yesterday they were not available. Can we have call today(Nov 3, 7.45PM to 8.15PM IST)? I Will schedule a meeting. works for you?
@icy-agent-73298 What would be the best time for you?
i
Yes sure works for me .(pmahindrakar@union.ai)
d
@icy-agent-73298 Hi, Did you get an invite from me?
i
Yes I did
d
okay.. Thank you 🙂
Thank you so much @icy-agent-73298 That was very helpful 🙂
👍 1
i
Let me know if you face any issues.
d
Sure...👍
Hi @icy-agent-73298 I have got clientIds and had setup them for Flyteconsole, Flytepropeller, Flytectl . Flytescheduler pod is still having a problem. I am getting below error in flytescheduler-check container. Do we need to have this grant type [authorization_code] assigned to Propeller/scheduler clientid? I dont see the list of grant types required for propeller clientId anywhere in the documentation. What are the other grant types required here?
Copy code
panic: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"unauthorized_client","error_description":"The client is not authorized to use the provided grant type. Configured grant types: [authorization_code]."}
i
Hi @delightful-lion-57360, this flytepropeller which is shared with scheduler should be using the same grant types as ones for flyteconsole. i.e client credentials and authorization code . Attaching an image from okta config Yeah i think these are not documented currently but would be great if you can help file a doc issue for it and also if possible help with a PR on it once you have things working .
d
@icy-agent-73298 Now, I am getting this error. What could be the reason? I have set the credentials as per the doc.
Copy code
panic: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken
i
Can you check what error do you see in the admin logs which would provide you more details
We can probably get on a call to debug this if that’s easier
d
I dont see any issues with admin pods. Here is the log from it. Sure, call would be really helpful. What time works for you? @icy-agent-73298
Copy code
]$ kubectl logs flyteadmin-f884d7f97-b99mb -n flyte
time="2022-11-20T23:58:22Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/clusters.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
{"json":{},"level":"warning","msg":"stow configuration section missing, defaulting to legacy s3/minio connection config","ts":"2022-11-20T23:58:23Z"}
i
Can we get on a call now if you are available.
d
Sure, let me ping you the zoom link
👍 1
@icy-agent-73298 Thanks for joining the call and helping me to resolve the issue. We have added allowedAudience in the AuthServer which resolved the issue.
Since pods are up. I am testing by registering the flyte workflows with this new auth setup
@icy-agent-73298 Thank for helping me out in the personal chat. The below config works in my local
Copy code
admin:
  endpoint: dns:///flyte.dev.xxx.xxx.com
  authType: ClientSecret
  clientId: <id>
  clientSecretLocation: /home/kkanagar/.flyte/client_secret
  insecure: false
  scopes: [ "all" ]
The workflow is registered in the Flyte and able to see the workflow in the console but the status is showing UNKNOWN.
This is the log from admin pod.
Copy code
{"json":{"exec_id":"fb21c31a2c48148b4b51","src":"execution_manager.go:381"},"level":"warning","msg":"Failed to fetch override values when assigning task resource default values for [resource_type:WORKFLOW project:\"examples\" domain:\"hbomax\" name:\"flyte.workflows.hi_world.my_wf\" version:\"3PMLBzY2tBl5bHSxXauKHQ==\" ]: Resource [{Project:examples Domain:hbomax Workflow:flyte.workflows.hi_world.my_wf LaunchPlan: ResourceType:TASK_RESOURCE}] not found","ts":"2022-11-22T21:27:44Z"}
{"json":{"exec_id":"fb21c31a2c48148b4b51","src":"execution_manager.go:385"},"level":"debug","msg":"Assigning task requested resources for [resource_type:WORKFLOW project:\"examples\" domain:\"hbomax\" name:\"flyte.workflows.hi_world.my_wf\" version:\"3PMLBzY2tBl5bHSxXauKHQ==\" ]","ts":"2022-11-22T21:27:44Z"}
{"json":{"src":"queues.go:43"},"level":"debug","msg":"refreshing execution queues","ts":"2022-11-22T21:27:44Z"}
{"json":{"exec_id":"fb21c31a2c48148b4b51","src":"queues.go:73"},"level":"warning","msg":"Failed to fetch override values when assigning execution queue for [{ResourceType:WORKFLOW Project:examples Domain:hbomax Name:flyte.workflows.hi_world.my_wf Version:3PMLBzY2tBl5bHSxXauKHQ== XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] with err: Resource [{Project:examples Domain:hbomax Workflow:flyte.workflows.hi_world.my_wf LaunchPlan: ResourceType:EXECUTION_QUEUE}] not found","ts":"2022-11-22T21:27:44Z"}
{"json":{"exec_id":"fb21c31a2c48148b4b51","src":"queues.go:109"},"level":"info","msg":"found no matching queue for [{ResourceType:WORKFLOW Project:examples Domain:hbomax Name:flyte.workflows.hi_world.my_wf Version:3PMLBzY2tBl5bHSxXauKHQ== XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]","ts":"2022-11-22T21:27:44Z"}
{"json":{"exec_id":"fb21c31a2c48148b4b51","src":"execution_manager.go:529"},"level":"info","msg":"getting the workflow execution config from application configuration","ts":"2022-11-22T21:27:44Z"}
{"json":{"src":"handlers.go:209"},"level":"debug","msg":"Found existing metadata Bearer eyJraWQiOiJPcTYzbzFRWUlDaDJVTExmTFM4bDRyRlRRbFQzcTRxNGljcW0xZVVDS1JFIiwiYWxnIjoiUlMyNTYifQ.eyJ2ZXIiOjEsImp0aSI6IkFULi1jdUwwMGl4eWRLNVFNZzNqSWJwbGZOakdRd2g0aGJtUTF1bzVEY1RaLWMiLCJpc3MiOiJodHRwczovL3R3ZGV2Lm9rdGFwcmV2aWV3LmNvbS9vYXV0aDIvZGVmYXVsdCIsImF1ZCI6ImFwaTovL2RlZmF1bHQiLCJpYXQiOjE2NjkxNTI0NjMsImV4cCI6MTY2OTE1NjA2MywiY2lkIjoiMG9hMWY0djhvZGs5OE5peFUwaDgiLCJzY3AiOlsiYWxsIl0sInN1YiI6IjBvYTFmNHY4b2RrOThOaXhVMGg4In0.Lw6aEcITlfkkul3zTbkeM0N27s3dmIA8JcDrheQ1iruBNsIVLzoIB_L7ezeSlzK0yDlh43RUbX1gGEQc_ELOuveeCv1iU-GOR5UoiaacdF4t5c7aOm5SGX5Hchmdh-MIe4dAsiUOIN1nABNrN8E44I9FsEB0HxWd-JsQx7W7-vs_KSf8aB0orooupN9xFEOlPImMyjNZpdFBzbTmRAVgxpmbaPk4-4nUy4CjejVFi1jUsD_t4Q-84fuhctzm0XdSUUnPGgUnFOId95nH_N3VGLRyhQd_a9Z2nskwLCA-_Er_gupQJfksFEC-kZbr9Yql0Z4ITauL5zE0OpKiH18RHQ","ts":"2022-11-22T21:27:44Z"}
{"json":{"src":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-11-22T21:27:44Z"}
{"json":{"src":"handlers.go:193"},"level":"debug","msg":"gRPC server info in logging interceptor [0oa1f4v8odk98NixU0h8]method [/flyteidl.service.AdminService/GetExecution]\n","ts":"2022-11-22T21:27:44Z"}
{"json":{"src":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"handlers.go:247"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"token.go:104"},"level":"debug","msg":"Could not retrieve id token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"handlers.go:247"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"token.go:104"},"level":"debug","msg":"Could not retrieve id token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2022-11-22T21:27:49Z"}
{"json":{"src":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-11-22T21:27:59Z"}
{"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-11-22T21:27:59Z"}
i
Can you check the propeller logs, seems its not able to report the progress when talking to admin and hence giving you those auth errors. Check if its due to same scope 
all
not provided in the propeller config map. If you have configured the scopes in the admin configmap then we don’t have to explicity set it .
Copy code
thirdPartyConfig:
            flyteClient:
              clientId: <flytectl-client-id>
              redirectUri: <http://localhost:53593/callback>
              scopes:
              - all
This section mentions that scope in the thirdPartyConfig https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#id1 Can you check if you have configured it in your flyteadmin-base config
d
@icy-agent-73298 I have ensured that the scopes are set fine. This is the below error keep coming from propeller also similar error from admin pod as well.
Copy code
E1123 17:01:25.016586       1 workers.go:102] error syncing 'examples-hbomax/f91bb33abf49a4eada70': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]
E1123 17:01:25.017858       1 workers.go:102] error syncing 'examples-hbomax/f8ef2c9513b334b17b45': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]
• pods are up • Able to register the workflow to flyte But the workflow is not moving to running state. it is still in unknown.. if we check admin & propeller logs, seeing the above error.
@icy-agent-73298 Thank you so much for your time today over a call to fix the issue faced. We are able to register and successfully run the workflow after redeploying the flytepropeller.
a
@delightful-lion-57360 we are experiencing a similar issue. What did you have to change?
186 Views
Open in Slack
PreviousNext
Powered by