https://flyte.org logo
#announcements
Title
# announcements
h

Hampus Rosvall

08/18/2022, 5:47 AM
Hey, I am trying to use the
clientSecret
authflow to be able to build and register workflows in our GitHub Actions CI builds. I looked at the CI docs here, and getting the following error both locally and in our CI when I run the same command. Has anyone experienced anything similar? Flytectl config available in comments
Copy code
☁  ~  flytectl get projects
Error: rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}
Copy code
admin:
  endpoint: dns:///domain
  authType: ClientSecret
  clientId: theClientId 
  clientSecretLocation: /path/to/secret # not base64 encoded 
  insecure: false 
logger:
  show-source: true
  level: 1
s

Samhita Alla

08/18/2022, 6:24 AM
The flow seems right to me. Can you double check if the client ID and secret are correct?
h

Hampus Rosvall

08/18/2022, 6:51 AM
They are correct! Using the same ones in the admin. Should I do any encoding?
p

Prafulla Mahindrakar

08/18/2022, 7:10 AM
Hi @Hampus Rosvall, Can you check your admin config and if the clientId matches what you are passing. Specifically this section and also from the log seems authServerType might not have been set in your case and defaulting to SelfAuth
Copy code
appAuth:
        authServerType: External
        externalAuthServer:
          baseUrl: https://<provider-url>/oauth2/<auth-server-id>
        thirdPartyConfig:
          flyteClient:
            clientId: <client-id>
            redirectUri: <http://localhost:53593/callback>
            scopes:
            - all
h

Hampus Rosvall

08/18/2022, 7:27 AM
Hey, the admin pod is crashlooping after I applied those changes:
Copy code
☁  terraform [main] ⚡ k logs flyteadmin-7fc47f776d-ddn4n
Defaulted container "flyteadmin" out of: flyteadmin, run-migrations (init), seed-projects (init), sync-cluster-resources (init), generate-secrets (init)
time="2022-08-18T07:24:48Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/clusters.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/logger.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
{"json":{},"level":"debug","msg":"Config section [notifications] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [domains] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [externalevents] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [queues] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [qualityofservice] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [task_resources] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [plugins] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [plugins.catalogcache] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [plugins.k8s] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [remotedata] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [namespace_mapping] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [database] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [auth] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [server] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [scheduler] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [cloudevents] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [secrets] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"info","msg":"setting metrics keys to [project domain wf task phase tasktype runtime_type runtime_version app_name]","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"info","msg":"Serving Flyte Admin Insecure","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"info","msg":"Starting profiling server on port [10254]","ts":"2022-08-18T07:24:48Z"}
{"json":{},"level":"error","msg":"Error creating resource server 404 Not Found: ","ts":"2022-08-18T07:24:48Z"}
Error: 404 Not Found:
Usage:
  flyteadmin serve [flags]
Your suggestions seems related to https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#oauth2-authorization-server, and are not outlined in CI section. Or am I missing something?
p

Prafulla Mahindrakar

08/18/2022, 7:39 AM
May be we are missing these things in the CI docs . Can you share your entire auth section of admin config. Do the baseUrl in appAuth match one thats in userAuth
h

Hampus Rosvall

08/18/2022, 7:44 AM
Gotcha! No worries, sure one sec
Copy code
server.yaml: |
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          baseUrl: <https://login.microsoftonline.com/tenantId/v2.0>
        thirdPartyConfig:
          flyteClient:
            clientId: clientId
            redirectUri: <http://localhost:53593/callback>
            scopes:
            - offline
            - all
      authorizedUris:
      - <https://domain>
      - <http://flyteadmin:80>
      - <http://flyteadmin.flyte.svc.cluster.local:80>
      userAuth:
        openId:
          baseUrl: <https://login.microsoftonline.com/tenantId/v2.0>
          clientId: clientId
          scopes:
          - profile
          - openid
Everything is matching, just masked out the actual Ids!
p

Prafulla Mahindrakar

08/18/2022, 7:50 AM
Does this open up on your browser
Copy code
<https://login.microsoftonline.com/tenantId/v2.0/.well-known/oauth-authorization-server>
h

Hampus Rosvall

08/18/2022, 7:52 AM
No it doesn’t, only
openid-configuration
suffix works
It returns 302
p

Prafulla Mahindrakar

08/18/2022, 7:55 AM
Ok can you add
Copy code
appAuth:
        authServerType: External
        externalAuthServer:
          baseUrl: <https://login.microsoftonline.com/tenantId/v2.0>
          metadataUrl: .well-known/openid-configuration
And check
h

Hampus Rosvall

08/18/2022, 7:56 AM
Sure
p

Prafulla Mahindrakar

08/18/2022, 8:00 AM
Also if you can search for jwks_uri/jwksUri in .well-know config if it exists
h

Hampus Rosvall

08/18/2022, 8:03 AM
jwks_uri is there!
p

Prafulla Mahindrakar

08/18/2022, 8:04 AM
cool . then it should not fail there any more with the new config update.
h

Hampus Rosvall

08/18/2022, 8:05 AM
Trying now, just restarting pods
The scopes all and offline are not supported my Microsoft
Copy code
☁  terraform [main] ⚡ flytectl get projects
Error: rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_scope","error_description":"AADSTS1002012: The provided value for scope offline all is not valid
p

Prafulla Mahindrakar

08/18/2022, 8:09 AM
Ok . Seems there is offline_access instead for ms
h

Hampus Rosvall

08/18/2022, 8:10 AM
Yeah, in your docs it says that the all scope is required. I could try only the offline_access scope
p

Prafulla Mahindrakar

08/18/2022, 8:12 AM
openid, email would also be required
It would be used for checking the logged in user who initiated the request
h

Hampus Rosvall

08/18/2022, 8:18 AM
Right, now I get this error
Copy code
☁ ⚡ flytectl get projects
Error: rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_scope","error_description":"AADSTS1002012: The provided value for scope openid email offline_access is not valid. Client credential flows must have a scope value with /.default suffixed to the resource identifier
h

Hampus Rosvall

08/18/2022, 9:09 AM
Sure, will check it out and let you know how it goes! 🙂 Thanks for your support
Did not work to use that scope either..
p

Prafulla Mahindrakar

08/18/2022, 1:45 PM
Strange. what is the scope you added and also is your environment up with flytepropeller being able to communicate with admin .
h

Hampus Rosvall

08/18/2022, 3:22 PM
how would I verify that communication?
p

Prafulla Mahindrakar

08/18/2022, 3:27 PM
Are you able to lunch any workflow from the flyteconsole UI and see the state changes being updated for the task
i used this scope https://graph.microsoft.com/.default
That seems to be an example scope
h

Hampus Rosvall

08/18/2022, 3:34 PM
Yes I can run workflows from UI/flytectl
And see graph of nodes, going from running -> completed etc
Copy code
i used this scope <https://graph.microsoft.com/.default>
That seems to be an example scope
Right, I really just know the oidc/oauth2 flows on a high level so I just tried whatever was there 😄
p

Prafulla Mahindrakar

08/18/2022, 3:54 PM
Thats strange. If propeller is able to do it so why can’t flytectl , Can you reuse flytepropeller’s client_id and client_secret and recheck in your original setup where things worked from UI .Change the clientId under flyteClient: to what you have configured for propeller
Apologize but this is our first experience with Azure AD so not sure whats going on.
h

Hampus Rosvall

08/18/2022, 4:28 PM
No need to apologise haha, really appreciate the help!
I will check tomorrow :)
👍 1
Copy code
server.yaml: |
    auth:
      appAuth:
        thirdPartyConfig:
          flyteClient:
            clientId: flytectl
            redirectUri: <http://localhost:53593/callback>
            scopes:
            - all
            - offline
      authorizedUris:
      - <https://domain>
      - <http://flyteadmin:80>
      - <http://flyteadmin.flyte.svc.cluster.local:80>
      userAuth:
        openId:
          baseUrl: <https://login.microsoftonline.com/tenantId/v2.0>
          clientId: clientId
          scopes:
          - profile
          - openid
This is my config currently. Why would I change from our IdP clientId to use Flytepropeller? Not quite sure I am following
p

Prafulla Mahindrakar

08/22/2022, 5:59 AM
Hi @Hampus Rosvall can we get on a call to resolve this.
h

Hampus Rosvall

08/22/2022, 11:36 AM
Hey! Sure we can do that, what’s your availability?
p

Prafulla Mahindrakar

08/22/2022, 11:50 AM
I am available now . we can do google meet if you are too
I am available for the next 2 hours .Ping me whenever you are
182 Views