Hi guys, I am trying to deploy Flyte to our Kubernetes stack but it seems like our FlyteScheduler deployment does not seem to want to start. I keep noticing this error in the Flyte Scheduler logs.

Error: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 401 Unauthorized

Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}

This is the log from FlyteAdmin logs:

{
  "json": {
    "src": "token.go:37"
  },
  "level": "info",
  "msg": "Error occurred in NewAccessRequest: invalid_client",
  "ts": "2022-03-30T19:38:33Z"
}

My Flyte Scheduler configmap is listed below. Am I perhaps missing a step?

data:
  admin.yaml: |
    admin:
      clientId: ExternalODICclientID
      clientSecretLocation: /etc/secrets/flyte-secret-auth
      endpoint: flyteadmin.flyte.svc:81
      insecure: true
    event:
      capacity: 1000
      rate: 500
      type: admin
  db.yaml: |
    database:
      dbname: postgres
      host: 'postgres-postgresql.flyte.svc'
      passwordPath: /etc/db/pass.txt
      port: 5432
      username: xxxx
  logger.yaml: |
    logger:
      level: 4
      show-source: true
  server.yaml: |
    scheduler:
      metricsScope: 'flyte:'
      profilerPort: 10254

This is our Flyte Admin configmap:

auth:
  appAuth:
    openId:
      baseUrl: <https://ExternalODICbaseurl.com>
      clientId: OurODICClientID
      scopes:
      - profile
      - openid
      - email
    thirdPartyConfig:
      flyteClient:
        clientId: ExternalODICclientID
        redirectUri: <http://localhost:53593/callback>
        scopes:
        - offline
        - all
  authorizedUris:
  - <https://our.domain.com>
  userAuth:
    openId:
      baseUrl: <https://ExternalODICbaseurl.com>
      clientId: ExternalODICclientID
      scopes:
      - profile
      - openid
      - email
  authorizedUris:
  - <https://our.domain.com>

cc @Haytham Abuelfutuh

@Johnson Huynh hey - you guys are running auth right? using google or something as an idp?

Yep that's correct, we're currently using google

There is no

openId

section under `appAuth`… that doesn’t look right…

and something else is terminating the ssl which is why you’re using insecure?

Yep that is correct

So if you are using Google IdP (And not Google Cloud Identity), you can’t use use Google IdP to secure your Flyte service for Apps… basically you can only do OAuth2 with Google’s services not with your own service (Flyte in this case).. But you CAN use Oidc with Google as well as other services (Why the limitation is beyond me…)

oh..that would explain why google wasn't presented there

To overcome that, FlyteAdmin ships with its own OAuth Authorization Server… we call it `selfAuthServer`… and unless you specify anything under

appAuth

config section, that’s what will be used for handling app auth…

ah ok I will do that, thanks again for the help!

I believe this points to a missed step here regarding flyteScheduler: https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#apply-configuration We need to make sure it’s properly configured and then add a step there to roll out its deployment as well…

Also for the flyte-schedule configmap should I remove the clientid and clientsecret?

You do need clientid and secret… can you check out flytepropeller configmap? the connection part should look identitical…

Ah I see, it uses the default "flytepropeller" id

cc @Sandra Youssef / @Alekhya can we help ensure that this disucssion gets somehow clearly understood through the docs or a discussion?