I m able to deploy the `flyte binary` chart on EKS When I tr Flyte #flyte-deployment

I'm able to deploy the `flyte-binary` chart on EKS...

Cody Scandore

08/15/2023, 1:48 PM

I'm able to deploy the

flyte-binary

chart on EKS. When I try to add

auth

the container goes to CrashLoopBackoff.

Copy code

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  6m49s                  default-scheduler  Successfully assigned flyte/flyte-flyte-binary-6479b586cc-sb75s to ip-192-xxx-xx-xxx.us-west-2.compute.internal
  Normal   Pulled     6m48s                  kubelet            Container image "postgres:15-alpine" already present on machine
  Normal   Created    6m48s                  kubelet            Created container wait-for-db
  Normal   Started    6m48s                  kubelet            Started container wait-for-db
  Normal   Started    6m47s                  kubelet            Started container gen-admin-auth-secret
  Normal   Created    6m47s                  kubelet            Created container gen-admin-auth-secret
  Normal   Pulled     6m47s                  kubelet            Container image "<http://cr.flyte.org/flyteorg/flyte-binary-release:v1.5.0|cr.flyte.org/flyteorg/flyte-binary-release:v1.5.0>" already present on machine
  Normal   Started    6m45s                  kubelet            Started container flyte
  Warning  Unhealthy  6m18s (x3 over 6m38s)  kubelet            Liveness probe failed: Get "<http://192.168.37.60:8088/healthcheck>": dial tcp 192.168.37.60:8088: connect: connection refused
  Normal   Killing    6m18s                  kubelet            Container flyte failed liveness probe, will be restarted
  Normal   Pulled     5m48s (x2 over 6m45s)  kubelet            Container image "<http://cr.flyte.org/flyteorg/flyte-binary-release:v1.5.0|cr.flyte.org/flyteorg/flyte-binary-release:v1.5.0>" already present on machine
  Normal   Created    5m48s (x2 over 6m45s)  kubelet            Created container flyte
  Warning  Unhealthy  107s (x52 over 6m44s)  kubelet            Readiness probe failed: Get "<http://192.168.37.60:8088/healthcheck>": dial tcp 192.168.37.60:8088: connect: connection refused

Copy code

...
{"json":{"src":"composite_workqueue.go:98"},"level":"debug","msg":"Dynamically configured batch size [-1]","ts":"2023-08-15T13:46:21Z"}
{"json":{"src":"composite_workqueue.go:129"},"level":"debug","msg":"Exiting SubQueue handler batch round","ts":"2023-08-15T13:46:21Z"}
{"json":{"src":"composite_workqueue.go:88"},"level":"debug","msg":"Subqueue handler batch round","ts":"2023-08-15T13:46:22Z"}
{"json":{"src":"composite_workqueue.go:98"},"level":"debug","msg":"Dynamically configured batch size [-1]","ts":"2023-08-15T13:46:22Z"}
{"json":{"src":"composite_workqueue.go:129"},"level":"debug","msg":"Exiting SubQueue handler batch round","ts":"2023-08-15T13:46:22Z"}

David Espejo (he/him)

08/15/2023, 3:00 PM

can you share the anonymized

values

you're using?

Cody Scandore

08/15/2023, 3:05 PM

Copy code

auth:
    enabled: true
    oidc:
      baseUrl: <https://login.microsoftonline.com/><my-client-id>/oauth2/v2.0/authorize
      clientId: 0xxxxx5-6xxx-4xxx-bxxx-3cxxxxxx
      clientSecret: 'Kxxxx~EqDxxxxxxxxxxxxxlxx~.Ddxxx'
    internal:
      clientSecret: 'monkeyJAX32'
      clientSecretHash: 'bW9ua2V5SkFYMzIK'
    authorizedUris:
      - <https://my-webapp.com>

David Espejo (he/him)

08/15/2023, 3:19 PM

is the

clientSecretHash

here dummy data? otherwise, I get a very different output for that

clientSecret

Cody Scandore

08/15/2023, 3:24 PM

They're real values. I used bcrypt but I had some errors launching the

gen-admin-auth-secret

container, along the lines of

Copy code

1 error(s) decoding:

* error decoding 'appAuth.selfAuthServer.staticClients[flytepropeller].client_secret': illegal base64 data at input byte 80

goroutine 1 [running]:
main.main()
        /flyteorg/build/cmd/main.go:13 +0x85

This value was generated from base64 on linux

Copy code

echo 'monkeyJAX32' | base64

and the deployment seemed (?) to progress further along.

David Espejo (he/him)

08/15/2023, 3:38 PM

I got the following using bcrypt:

JDJiJDA2JEp2dldEYk1ZZDB6ZWJJVnozRWIvM2VzcDE0NFJtRHh1QVdJakJLQzFyZDVMTjNQSGFlWkF1

Cody Scandore

08/15/2023, 3:47 PM

Okay, it seems like it's still crash looping with that value

Copy code

Normal   Killing    92s                 kubelet            Container flyte failed liveness probe, will be restarted
  Normal   Pulled     62s (x2 over 116s)  kubelet            Container image "<http://cr.flyte.org/flyteorg/flyte-binary-release:v1.5.0|cr.flyte.org/flyteorg/flyte-binary-release:v1.5.0>" already present on machine
  Normal   Created    62s (x2 over 116s)  kubelet            Created container flyte
  Warning  Unhealthy  62s (x9 over 115s)  kubelet            Readiness probe failed: Get "<http://192.168.37.60:8088/healthcheck>": dial tcp 192.168.37.60:8088: connect: connection refused
  Normal   Started    61s (x2 over 116s)  kubelet            Started container flyte

David Espejo (he/him)

08/15/2023, 3:59 PM

can you try with

baseUrl: <https://login.microsoftonline.com/><my-client-id>/oauth2/

Cody Scandore

08/15/2023, 4:02 PM

Same, failing readiness. Not much helpful information in the pod logs, as far as I can tell.

David Espejo (he/him)

08/15/2023, 4:41 PM

has it worked in the past?

Cody Scandore

08/15/2023, 4:41 PM

Cody Scandore

08/15/2023, 4:41 PM

Everything works well without

auth

, fails with auth enabled.

9 Views

Open in Slack

Previous Next