https://flyte.org logo
Join Slack
Powered by
Hello. I am trying to configure authorization and ...
# flyte-support
a
Hello. I am trying to configure authorization and running into this error: 
flyte {"json":{"src":"start.go:58"},"level":"fatal","msg":"Flyte native scheduler failed to start due to async future was canceled"
Content of `values.yaml`:
Copy code
configuration:
  auth:
    enabled: true
    enableAuthServer: false
    oidc:
      baseUrl: "<https://flyte.company.com>"
      clientId: "flyte-auth"
      clientSecret: "xxx-client-secret-xxx"
    internal:
      clientId: flytepropeller
      clientSecret: "yyyyyby0fazfe0557mmlcae49obay4uojkkqgvy051llbkygrzzzzz"
      clientSecretHash: "JiOVJrSnBDSS5md2UzWGNBejJlWE9abHpYT1JRcFhPa0FSZnBuL"
    flyteClient:
      clientId: flytectl
      redirectUri: <http://localhost:53593/callback>
      scopes:
        - all
      audience: ""
    authorizedUris:
      - <https://flyte.company.com/console>
    clientSecretsExternalSecretRef: ""
We use DEX as auth provider.
Copy code
staticClients:
    - id: flyte-auth
      name: flyte-auth
      secret: xxx-client-secret-xxx
      redirectURIs:
        - <http://flyte.company.com/console/callback>
        - <https://flyte.company.com/console/callback>
      # public: true
      trustedPeers:
        - oauth2-proxy
Ingress:
Copy code
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  name: flyte-ingress
  namespace: flyte
  labels:
    release: flyte
    heritage: Helm
spec:
  tls:
    - hosts:
        - "*.<http://company.com|company.com>"
      secretName:
  ingressClassName: nginx
  rules:
    - host: "<http://flyte.company.com|flyte.company.com>"
      http:
        paths:
          - path: /console
            pathType: ImplementationSpecific
            backend:
              service:
                name: flyte-binary-http
                port:
                  name: http
Based on this comment by @average-finland-92144, I also tried to configure the 
authorizedUris
entry but with no luck. How do I fix this?
a
hey @aloof-tailor-52290, thanks for sharing your config. I haven't used DEX but find some aspects of your config worth inspecting: 1. Is this intended to use DEX as the External auth server correct? 2. In any case, make sure the 
baseUrl
is the one provided by DEX to your OIDC client. The ingress host is not typically the one to use here 3. If I take your 
clientSecret
for propeller and calculate the hash with 
bcrypt
I get a different value than what you have in this config 4. In the 
authorizedUri
, use the base URL of your Ingress host, without the 
/console
path 5. Make sure the 
flyteclient
section lies inside the 
inline
section as indicated by the docs 6. Also the 
redirectURI
in DEX shouldn't include the 
/console
path 7. 
flyteclient
needs the 
offline
scope too 8. I don't see the 
userAuth
config here, needed for OIDC Let's verify this and hopefully we'll isolate the issue. That error message happens frequently when the 
flytescheduler-check
initContainer, one of the first "clients" that tries the connection to flyteadmin, fails establishing an authenticated session
a
Hi @average-finland-92144. Sorry for the late response. Actually this was taking time so we decided to deploy it without auth so the devs could proceed with evaluating it. I thought I will try your suggestion by installing flyte in a different ns but seems it's pointing to the same old flyte which is currently running without auth enabled. Are these installations in different ns using the same etcd? Is it possible to have two flyte set up in the same k8s cluster?
@average-finland-92144, I spent some time reading about oauth and dex. Based on that as well as your suggestions, I have made some changes (also, ingress used now has been generated by the helm chart). I think I also overlooked the 
inline
section. Thanks for pointing it out. ingress-flyte-binary-http.yaml
Copy code
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/app-root: /console
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  labels:
    app.kubernetes.io/instance: flyte-binary
  name: flyte-binary-http
  namespace: flyte
spec:
  ingressClassName: nginx
  rules:
  - host: flyte.us.company.com
    http:
      paths:
      - backend:
          service:
            name: flyte-binary-http
            port:
              number: 8088
        path: /console
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-http
            port:
              number: 8088
        path: /console/*
        pathType: ImplementationSpecific
      ...
      ...
      ...
      - backend:
          service:
            name: flyte-binary-http
            port:
              number: 8088
        path: /.well-known
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-http
            port:
              number: 8088
        path: /.well-known/*
        pathType: ImplementationSpecific
      ...
      ...
      ...
      - backend:
          service:
            name: flyte-binary-http
            port:
              number: 8088
        path: /callback
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-http
            port:
              number: 8088
        path: /callback/*
        pathType: ImplementationSpecific
      ...
      ...
      ...
      - backend:
          service:
            name: flyte-binary-http
            port:
              number: 8088
        path: /oauth2
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-http
            port:
              number: 8088
        path: /oauth2/*
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - '*.us.company.com'
DEX config:
Copy code
connectors:
    - id: ldap-ipa
      name: ldap-ipa
      type: ldap
      config:
        bindDN: uid=dexaccount,cn=accounts,cn=etc,dc=company,dc=com
        bindPW: h9d34ac665
        groupSearch:
        ...
        ...
        userSearch:
            PreferredUsernameAttr: uid
            baseDN: cn=users,cn=accounts,dc=company,dc=com

issuer: <https://dex.us.company.com/>

oauth2:
    responseTypes:
        - code
        - token
        - id_token
    skipApprovalScreen: true
    
staticClients:
    - id: flyte-auth
      name: flyte-auth
      secret: abcdefghijklmnopqr123456789
      redirectURIs:
        - <http://flyte.us.company.com/callback>
        - <https://flyte.us.company.com/callback>
      trustedPeers:
        - oauth2-proxy
values.yaml
Copy code
configuration:
  # auth Specify configuration for Flyte authentication
  auth:
    enabled: true
    enableAuthServer: false
    oidc:
      baseUrl: "<https://dex.us.company.com/>"
      # clientId Flyte application client ID
      clientId: "flyte-auth"
      clientSecret: "abcdefghijklmnopqr123456789"
    internal:
      # clientId Client ID for internal authentication - set to flytepropeller or external auth server
      clientId: flytepropeller
      clientSecret: "61ttblq6oyhby0fazfe0557mmlcae49obay4uojkkqgvy051llbkygrz335uoq0g"
      clientSecretHash: "JDJiJDA2JEJiOVJrSnBDSS5md2UzWGNBejJlWE9abHpYT1JRcFhPa0FSZnBuLzlrS0tSTFBWQlFUdi9L"
    # flyteClient:
    #   clientId: flytectl
    #   redirectUri: <http://localhost:53593/callback>
    #   scopes:
    #     - all
    authorizedUris:
    - <https://flyte.us.company.com>

  # inline Specify additional configuration or overrides for Flyte, to be merged with the base configuration
  inline:
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          baseUrl: <https://dex.us.company.com/>
          metadataUrl: .well-known/oauth-authorization-server
        thirdPartyConfig:
          flyteClient:
            # Use the clientID generated by your IdP for the `flytectl` app registration
            clientId: <flytectl-client-id>
            redirectUri: <http://localhost:53593/callback>
            scopes:
            - offline
            - all
      userAuth:
        openId:
          baseUrl: <https://dex.us.company.com/>
          scopes:
          - profile
          - openid
        # - offline_access # Uncomment if your IdP supports issuing refresh tokens (optional)
        # Use the client ID and secret generated by your IdP for the first OIDC registration in the "Identity Management layer : OIDC" section of this guide
          clientId: <oidc-clientId>
Few things in 
inline
block are still to be populated because I am not too sure about them. 1. I generated random password and corresponding hash for flytepropeller (configuration.auth.internal.clientId) using this link. I hope that's right. 2. Not sure what the value of 
configuration.inline.auth.appAuth.externalAuthServer.metadataUrl
should be but I set it to 
.well-known/oauth-authorization-server
based on the ingress generated by the helm chart which has a backend path 
/.well-known
. Just guessing here. If you know what the value should be, please let me know. 3. How can I determine the value of 
configuration.inline.auth.appAuth.thirdPartyConfig.flyteClient.clientId
? DEX, as far as I know, doesn't provide any interface to generate client-id and secret. It's just some static value. 4. Will 
configuration.inline.auth.userAuth.openId.clientId
have the same value as 
configuration.auth.oidc.clientId
?
Also added some extra config to DEX above. It connects to FreeIPA (LDAP).
a
Are these installations in different ns using the same etcd? Is it possible to have two flyte set up in the same k8s cluster?
Yes, the supported format here is one Flyte deployment per Kubernetes cluster, which uses the underlying etcd instance. This is mainly due to the ownership on the 
flyteworkflow
CRD
1. Not sure if I'm missing something, but still I get a different hash when I use your 
configuration.auth.internal.clientSecret
value:
Copy code
python -c 'import bcrypt; import base64; print(base64.b64encode(bcrypt.hashpw("61ttblq6oyhby0fazfe0557mmlcae49obay4uojkkqgvy051llbkygrz335uoq0g".encode("utf-8"), bcrypt.gensalt(6))))'
b'JDJiJDA2JFliakc3S2MxcUllbFF2NTltN3RBOE96Z3kyeXN6NEkvTzdrQnVUMk1kWUNTajhTZC5FRG02'
2. Not sure how dex implements it, but as far as the RFC for auth servers metadata is concerned, the default URL should be: 
.well-known/oauth-authorization-server
3. IIUC, that's what you configure in dex's 
staticClients
section. Once you indicate a client id and secret there, your app is "registered". You would need then a separate client for 
flytectl
which, at least in Okta, is configured as a 
Native app
. It would cover not only flytectl but also pyflyte. Not sure how this is done in dex though
a
1. Not sure if I'm missing something, but still I get a different hash when I use your 
configuration.auth.internal.clientSecret
value:
I just generated a random string using [this](https://www.browserling.com/tools/random-string) link and then ran the command you pasted (
pip install bcrypt && python -c 'import bcrypt; import base64; print(base64.b64encode(bcrypt.hashpw("<your-random-password>".encode("utf-8"), bcrypt.gensalt(6))))'
). I see everytime I run this command, it generates a new value. Any tips?
Regarding dex's 
staticClients
section, I also had the same feeeling to create another block. I will test that. Thanks
Here's the updated DEX config. Only showing the updated portion. Not sure whether for the second client-id (flytectl-auth), 
redirectURIs
will remain the same as that of the first one.
Copy code
staticClients:
    - id: flyte-auth
      name: flyte-auth
      secret: abcdefghijklmnopqr123456789
      redirectURIs:
        - <http://flyte.us.company.com/callback>
        - <https://flyte.us.company.com/callback>
      trustedPeers:
        - oauth2-proxy
    - id: flytectl-auth
      name: flytectl-auth
      secret: cyi92may466lf29ev8da4ubay02
      redirectURIs:
        - <http://flyte.us.company.com/callback>
        - <https://flyte.us.company.com/callback>
      # public: true
      trustedPeers:
        - oauth2-proxy
Also, what should be value for 
configuration.inline.auth.appAuth.thirdPartyConfig.flyteClient.redirectUri
?
Copy code
auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          baseUrl: <https://dex.us.company.com/>
          metadataUrl: .well-known/oauth-authorization-server
        thirdPartyConfig:
          flyteClient:
            clientId: flytectl-auth
            redirectUri: <http://localhost:53593/callback>
            scopes:
            - offline
            - all
With above values, the deployment had no effect. I don't get a DEX login page. I just land up on the project page (https://flyte.us.company.com/console/select-project)
a
so you have two clients created in DEX and two configured on Flyte right?
a
Yes
a
what should be value for 
configuration.inline.auth.appAuth.thirdPartyConfig.flyteClient.redirectUri
?
typically, you'd use 
<http://localhost:53593/callback>
like you seem to have it configured.
a
The way I generated the secret and hash, is that correct? If yes, any idea why it keeps generating different hashes each time?
a
It seems to be by design, I now see the same behavior
👍 1
a
Will configuration.inline.auth.userAuth.openId.clientId have the same value as configuration.auth.oidc.clientId?
Is this correct?
From this link, looks like my value is correct for 
configuration.inline.auth.userAuth.openId.clientId
I fixed one important config. The official link (https://docs.flyte.org/en/latest/deployment/configuration/auth_setup.html#apply-oidc-configuration) led me to earlier believe that a random secret had to be hashed. However, when I read this link (https://github.com/davidmirror-ops/flyte-the-hard-way/blob/main/docs/aws/11-upgrade-with-auth.md), it mentioned 
Take the Client Secret generated by Okta for the flytepropeller App Integration and generate a hash
so I registered a third client (
flytepropeller-client
), used its secret for hashing. values.yaml
Copy code
auth:
    enabled: true
    enableAuthServer: false
    oidc:
      baseUrl: "<https://dex.us.company.com/>"
      clientId: "flyteconsole-client"
      clientSecret: "gnkkl2f7aft7p9itho3emihymq1drfwdb3i9u3"
    internal:
      clientId: flytepropeller-client
      clientSecret: "sci07nkd4cj9zi3jyawmfwwmuaf5k8r593zdh"
      clientSecretHash: "...JDJiJDA2JEg4aHlvbEFvWEZDdWtWMkQ1WlhXRS5hWWlW..."
      audience: ""
    authorizedUris:
    - <https://flyte.us.company.com>
    clientSecretsExternalSecretRef: ""
  inline:
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          baseUrl: <https://dex.us.company.com/>
          metadataUrl: .well-known/oauth-authorization-server
        thirdPartyConfig:
          flyteClient:
            clientId: flytectl-client
            redirectUri: <http://localhost:53593/callback>
            scopes:
            - offline
            - all
      userAuth:
        openId:
          baseUrl: <https://dex.us.company.com/>
          scopes:
          - profile
          - openid
          clientId: flyteconsole-client
DEX
Copy code
- id: flyteconsole-client
      name: flyteconsole-client
      secret: gnkkl2f7aft7p9itho3emihymq1drfwdb3i9u3
      redirectURIs:
        - <http://flyte.us.company.com/callback>
        - <https://flyte.us.company.com/callback>
      # public: true
      trustedPeers:
        - oauth2-proxy
    - id: flytepropeller-client
      name: flytepropeller-client
      secret: sci07nkd4cj9zi3jyawmfwwmuaf5k8r593zdh
      redirectURIs:
        - <http://flyte.us.company.com/callback>
        - <https://flyte.us.company.com/callback>
      # public: true
      trustedPeers:
        - oauth2-proxy
    - id: flytectl-client
      name: flytectl-client
      secret: l8xpetybtfw3e1l56fknyvefylbii6qfzqlta2
      redirectURIs:
        - <http://flyte.us.company.com/callback>
        - <https://flyte.us.company.com/callback>
      # public: true
      trustedPeers:
        - oauth2-proxy
I really thought this would finally fix the issue and show me the dex login screen but seems some more work is needed 🙂
Copy code
flyte {"json":{"src":"schedule_executor.go:88"},"level":"info","msg":"Flyte native scheduler shutdown","ts":"2024-05-16T07:50:27Z"}                                                                             flyte {"json":{"src":"start.go:58"},"level":"fatal","msg":"Flyte native scheduler failed to start due to async future was canceled","ts":"2024-05-16T07:50:27Z"}                                               
Stream closed EOF for flyte/flyte-binary-6bd9c67fb8-9t7fk (flyte)
a
so if you go to console nothing happens? The error message above shows that there's an attempt to establish an authenticated session from flytescheduler to admin, which is expected but it fails
a
I can see the same projects page which shows up when auth is disabled. However, when I click on the Login button, it errors out.
48 Views
Open in Slack
PreviousNext
Powered by