https://flyte.org logo
j

Johnson Huynh

03/30/2022, 7:10 PM
Hi guys, I am trying to deploy Flyte to our Kubernetes stack but it seems like our FlyteScheduler deployment does not seem to want to start. I keep noticing this error in the Flyte Scheduler logs.
Copy code
Error: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 401 Unauthorized

Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}
This is the log from FlyteAdmin logs:
Copy code
{
  "json": {
    "src": "token.go:37"
  },
  "level": "info",
  "msg": "Error occurred in NewAccessRequest: invalid_client",
  "ts": "2022-03-30T19:38:33Z"
}
My Flyte Scheduler configmap is listed below. Am I perhaps missing a step?
Copy code
data:
  admin.yaml: |
    admin:
      clientId: ExternalODICclientID
      clientSecretLocation: /etc/secrets/flyte-secret-auth
      endpoint: flyteadmin.flyte.svc:81
      insecure: true
    event:
      capacity: 1000
      rate: 500
      type: admin
  db.yaml: |
    database:
      dbname: postgres
      host: 'postgres-postgresql.flyte.svc'
      passwordPath: /etc/db/pass.txt
      port: 5432
      username: xxxx
  logger.yaml: |
    logger:
      level: 4
      show-source: true
  server.yaml: |
    scheduler:
      metricsScope: 'flyte:'
      profilerPort: 10254
This is our Flyte Admin configmap:
Copy code
auth:
  appAuth:
    openId:
      baseUrl: <https://ExternalODICbaseurl.com>
      clientId: OurODICClientID
      scopes:
      - profile
      - openid
      - email
    thirdPartyConfig:
      flyteClient:
        clientId: ExternalODICclientID
        redirectUri: <http://localhost:53593/callback>
        scopes:
        - offline
        - all
  authorizedUris:
  - <https://our.domain.com>
  userAuth:
    openId:
      baseUrl: <https://ExternalODICbaseurl.com>
      clientId: ExternalODICclientID
      scopes:
      - profile
      - openid
      - email
  authorizedUris:
  - <https://our.domain.com>
k

Ketan (kumare3)

03/30/2022, 7:41 PM
cc @Haytham Abuelfutuh
y

Yee

03/30/2022, 8:03 PM
@Johnson Huynh hey - you guys are running auth right? using google or something as an idp?
j

Johnson Huynh

03/30/2022, 8:03 PM
Yep that's correct, we're currently using google
h

Haytham Abuelfutuh

03/30/2022, 8:04 PM
There is no
openId
section under `appAuth`… that doesn’t look right…
y

Yee

03/30/2022, 8:04 PM
and something else is terminating the ssl which is why you’re using insecure?
j

Johnson Huynh

03/30/2022, 8:05 PM
Yep that is correct
h

Haytham Abuelfutuh

03/30/2022, 8:08 PM
So if you are using Google IdP (And not Google Cloud Identity), you can’t use use Google IdP to secure your Flyte service for Apps… basically you can only do OAuth2 with Google’s services not with your own service (Flyte in this case).. But you CAN use Oidc with Google as well as other services (Why the limitation is beyond me…)
You can see under OAuth Authorization Server section, Google IdP isn’t listed…
j

Johnson Huynh

03/30/2022, 8:09 PM
oh..that would explain why google wasn't presented there
h

Haytham Abuelfutuh

03/30/2022, 8:10 PM
To overcome that, FlyteAdmin ships with its own OAuth Authorization Server… we call it `selfAuthServer`… and unless you specify anything under
appAuth
config section, that’s what will be used for handling app auth…
Take a look at the default config for that section here. You can’t register new clients programmatically, you can’t rotate keys programmatically but you can configure clients statically through flyteadmin config…
Out of box it ships with 3 apps flyte-cli, flytectl (both are “public” clients with no secrets) and flytepropeller with a base64 encoded secret right in the config (obviously not secure)
If you strictly follow the “Google” section in the auth setup guide and nothing more, you should be able to get scheduler and propeller talking with flyteAdmin using these statically configured clients…
j

Johnson Huynh

03/30/2022, 8:13 PM
ah ok I will do that, thanks again for the help!
h

Haytham Abuelfutuh

03/30/2022, 8:13 PM
I believe this points to a missed step here regarding flyteScheduler: https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#apply-configuration We need to make sure it’s properly configured and then add a step there to roll out its deployment as well…
j

Johnson Huynh

03/30/2022, 8:17 PM
Also for the flyte-schedule configmap should I remove the clientid and clientsecret?
And just leave endpoint only in the config?
Copy code
admin:
  clientId: OIDCclientID
  clientSecretLocation: /etc/secrets/client_secret
  endpoint: flyteadmin:81
  insecure: true
h

Haytham Abuelfutuh

03/30/2022, 8:18 PM
You do need clientid and secret… can you check out flytepropeller configmap? the connection part should look identitical…
j

Johnson Huynh

03/30/2022, 8:22 PM
Ah I see, it uses the default "flytepropeller" id
k

Ketan (kumare3)

03/30/2022, 8:48 PM
cc @Sandra Youssef / @Alekhya can we help ensure that this disucssion gets somehow clearly understood through the docs or a discussion?
👍 2
6 Views