Hey everyone slightly smiling face I m currently deploying a

Question

Hey everyone slightly smiling face I m currently deploying a flyte sandbox at my new company and am struggling with getting auth to work I followed guide Flyte is deployed I am redirected to the SSO login screen after that I can view the flyte console +1 I can also run `flytectl get projects` am redirected to the browser and then retrieve the projects `pyflyte register ` works as well So far so good When I manually launch a registered workflow from the console it doesn t start however `k n development describe ah9ghn272q7v22dbsvvd` gives ``` Status Failed Attempts 9 Message Workflow failed ErrorRecordingError failed to publish event caused by EventSinkError Error sending event caused by rpc error code = Unavailable desc = connection error desc = transport authentication handshake failed tls first record does not look like a TLS handshake Phase 0 Tasks resource type TASK project 3dod domain development name project workflows workflow deploy model version 2 ``` Details about config in thread A pointer in the right direction would be much appreciated

Fabio Grätz · Answer

I see similar logs in flytepropeller ```E1019 08 50 19 766619 1 workers go 102 error syncing development arzsbvcrdhd8jdbp2vdd Workflow failed ErrorRecordingError failed to publish event caused by EventSinkError Error sending event caused by rpc error code = Unavailable desc = connection error desc = transport authentication handshake failed tls first record does not look like a TLS handshake E1019 08 50 19 776908 1 workers go 102 error syncing development aprk6hb8gxdcj882njpf Workflow failed ErrorRecordingError failed to publish event caused by EventSinkError Error sending event caused by rpc error code = Unavailable desc = connection error desc = transport authentication handshake failed tls first record does not look like a TLS handshake json exec id ah9ghn272q7v22dbsvvd ns development res ver 1249185 routine worker 37 wf 3dod development project workflows workflow pipeline level warning msg Event recording failed Error EventSinkError Error sending event caused by rpc error code = Unavailable desc = connection error desc = transport authentication handshake failed tls first record does not look like a TLS handshake ts 2022 10 19T08 50 49Z ```

Fabio Grätz · Answer

This is the flyte admin base configmap ```apiVersion v1 data cluster resources yaml | cluster resources customData production projectQuotaCpu value 5 projectQuotaMemory value 4000Mi gsa value gsa production lt project gt staging projectQuotaCpu value 2 projectQuotaMemory value 3000Mi gsa value gsa staging lt project gt development projectQuotaCpu value 2 projectQuotaMemory value 3000Mi gsa value gsa development lt project gt refreshInterval 5m standaloneDeployment false templatePath etc flyte clusterresource templates db yaml | database dbname flyteadmin host 10 22 0 3 passwordPath etc db pass txt port 5432 username flyteadmin domain yaml | domains id development name development id staging name staging id production name production namespace config yaml | namespace mapping template domain remoteData yaml | remoteData scheme gcs signedUrls durationMinutes 3 server yaml | auth appAuth thirdPartyConfig flyteClient clientId flytectl redirectUri scopes offline all authorizedUris userAuth openId baseUrl clientId lt client id gt scopes profile openid flyteadmin eventVersion 2 metadataStoragePrefix metadata admin metricsScope flyte profilerPort 10254 roleNameKey testing host server grpcPort 8089 httpPort 8088 security allowCors true allowedHeaders Content Type allowedOrigins secure false useAuth true storage yaml | storage type stow stow kind google config json project id lt project gt scopes container company flyte bucket enable multicontainer false limits maxDownloadMBs 10 task resource defaults yaml | task resources defaults cpu 500m memory 500Mi storage 500Mi limits cpu 2 gpu 1 memory 1Gi storage 2000Mi kind ConfigMap metadata annotations flyte flyte creationTimestamp 2022 10 18T18 09 45Z labels flyte Helm flyteadmin flyte core v1 2 0 name flyte admin base config namespace flyte resourceVersion 780983 uid b5da3e3e 35d2 406b 953e ad568179a646```

Sören Brunk · Answer

It looks like flytepropeller auth against flyteadmin fails Do you use the embedded auth server

Fabio Grätz · Answer

Yes that is the plan Is it active by default

Fabio Grätz · Answer

```apiVersion v1 data admin yaml | admin clientId flytepropeller clientSecretLocation etc secrets client secret endpoint flyteadmin 81 insecure false event capacity 1000 rate 500 type admin cache yaml | cache max size mbs 1024 target gc percent 70 catalog yaml | catalog cache endpoint datacatalog 89 insecure true type datacatalog copilot yaml | plugins k8s co pilot image name flyte copilot start timeout 30s core yaml | manager pod application flytepropeller pod template container name flytepropeller pod template name flytepropeller template propeller downstream eval duration 30s enable admin launcher true gc interval 12h kube client config burst 25 qps 100 timeout 30s leader election enabled true lease duration 15s lock config map name propeller leader namespace flyte renew deadline 10s retry period 2s limit namespace all max workflow retries 50 metadata prefix metadata propeller metrics prefix flyte prof port 10254 queue batch size 1 batching interval 2s queue base delay 5s capacity 1000 max delay 120s rate 100 type maxof sub queue capacity 1000 rate 100 type bucket type batch rawoutput prefix workers 40 workflow reeval duration 30s webhook certDir etc webhook certs serviceName flyte pod webhook enabled plugins yaml | tasks task plugins default for task types container container container array k8s array sidecar sidecar enabled plugins container sidecar k8s array k8s yaml | plugins k8s default cpus 100m default env vars default memory 100Mi resource manager yaml | propeller resourcemanager type noop storage yaml | storage type stow stow kind google config json project id lt project gt scopes container company flyte bucket enable multicontainer false limits maxDownloadMBs 10 task logs yaml | plugins k8s array logs config stackdriver enabled true stackdriver logresourcename k8s container logs cloudwatch enabled false kubernetes enabled false stackdriver enabled true stackdriver logresourcename k8s container kind ConfigMap metadata annotations flyte flyte creationTimestamp 2022 10 18T18 09 47Z labels flyte Helm flyteadmin flyte core v1 2 0 name flyte propeller config namespace flyte resourceVersion 1245505 uid 4dd30035 f278 452c 8910 cc9bd4878c6b```

Sören Brunk · Answer

In previous versions it had a default secret set which caused a security issue because essentially anyone could connect with the `flytepropeller` client id using that default secret I think that was fixed so you might have to set that explicitly now

Sören Brunk · Answer

And yes I think it is active by default if you set `useAuth true` and don t explicitly configure an external auth server

Fabio Grätz · Answer

In the GCP values I took from the docs and slightly adapted with my values <https github com flyteorg flyte blob e40426e24e500b2031e9ba6d8ae28790c5a7fbd1 charts flyte core values yaml L290|this> section is missing Do you know whether it is related to the default secret you mentioned

Sören Brunk · Answer

Yes exactly that s the one

Sören Brunk · Answer

But if you change that here the change only applies to flytepropeller

Fabio Grätz · Answer

I need to ask another dumb question sorry ^^ Do I but a random made up secret there that both admin and propeller will then know or is this the same client id and secret from my oauth provider that is also used to configure the login screen

Sören Brunk · Answer

Flyteadmin does not use that config and still expects whatever is default in it s config used to be `foobar` as well hence the security issue

Sören Brunk · Answer

So in my case I had to override that part of the flyteadmin config and that is not really documented yet unfortunately let me see if I can find it

Sören Brunk · Answer

Regarding your question this is different from the client id you configure in Google

Sören Brunk · Answer

So you should put a random secret in there It is a shared secret for the internal oauth flow which is different than the browser login via google

Sören Brunk · Answer

Google login does not support an oauth client credentials flow that s why we can t use that here for the internal flytepropeller auth

Fabio Grätz · Answer

Ok got it will add the following to the helm values ```secrets adminOauthClientCredentials enabled true clientSecret lt random string gt clientId flytepropeller``` So this affects the communication between propeller and admin right Do you know how auth works between the other flyte services Console and admin work via the Google login I understand

Sören Brunk · Answer

Right I m not sure how i e flyteadmin talks to datacatalog but given working DB auth the communication between propeller and admin was the only thing that ever caused issues for us

Fabio Grätz · Answer

I added the secret section to the helm values but still get ```Status Failed Attempts 3 Message Workflow failed ErrorRecordingError failed to publish event caused by EventSinkError Error sending event caused by rpc error code = Unauthenticated desc = transport per RPC creds failed due to error oauth2 cannot fetch token 401 Unauthorized Response error invalid client error description Client authentication failed e g unknown client no client authentication included or unsupported authentication method Phase 0 Tasks ``` But I also didn t actively do anything to make admin aware of this secret

Sören Brunk · Answer

I found the relevant section to overwrite the default flyteadmin auth server config via helm ```configmap adminServer server httpPort 8088 grpcPort 8089 dataProxy upload storagePrefix upload security secure false useAuth true allowCors true allowedOrigins Accepting all domains for Sandbox installation allowedHeaders Content Type auth appAuth authServerType Self selfAuthServer accessTokenLifespan 30m0s authorizationCodeLifespan 5m0s claimSymmetricEncryptionKeySecretName claim symmetric key issuer oldTokenSigningRSAKeySecretName token rsa key old pem refreshTokenLifespan 1h0m0s staticClients flyte cli audience null grant types refresh token authorization code id flyte cli public true redirect uris response types code token scopes all offline access token flytectl audience null grant types refresh token authorization code id flytectl public true redirect uris response types code token scopes all offline access token flytepropeller audience null client secret lt your client secret hashed and base64 encoded gt grant types refresh token client credentials id flytepropeller public false redirect uris response types token scopes all offline access token```

Sören Brunk · Answer

flyterpropeller client secret is the relevant change Unfortunately it requires the secret as a base64 encoded hash I ll try to find how we did that hashing

Fabio Grätz · Answer

So I ran `echo n clientSecret | base64` which is what I would typically do for a k8s secret

Fabio Grätz · Answer

Still doesn t work sweat smile Flytescheduler is failing now `k n flyte logs f flytescheduler 6cbfb47f9f h76q4 c flytescheduler check` ``` storage limits maxDownloadMBs int Maximum allowed download size in MBs per call default 2 storage stow config stringToString Configuration for stow backend Refer to github flyteorg stow default storage stow kind string Kind of Stow backend to use Refer to github flyteorg stow storage type string Sets the type of storage to configure s3 minio local mem stow default s3 panic rpc error code = Unauthenticated desc = transport per RPC creds failed due to error oauth2 cannot fetch token 401 Unauthorized Response error invalid client error description Client authentication failed e g unknown client no client authentication included or unsupported authentication method goroutine 1 running main main go src github com flyteorg flyteadmin cmd scheduler main go 12 +0x85``` And my worklfow still has the status ```Status Failed Attempts 4 Message Workflow failed ErrorRecordingError failed to publish event caused by EventSinkError Error sending event caused by rpc error code = Unauthenticated desc = transport per RPC creds failed due to error oauth2 cannot fetch token 401 Unauthorized Response error invalid client error description Client authentication failed e g unknown client no client authentication included or unsupported authentication method Phase 0```

Sören Brunk · Answer

Yeah it also needs to be hashed in a specific way I ll try to find how we did it

Ketan (kumare3) · Answer

This should be documented I think it is Cc < Yee> < Eduardo Apolinario eapolinario > can you point how to create the client secret for Google

Fabio Grätz · Answer

< Eduardo Apolinario eapolinario > I think the Google IdP part is working correctly I can log into the console with SSO and can also register workflows with flytectl or pyflyte I think the problem happens with the communication between admin propeller and scheduler when I want to run a workflow My understanding is that there is a 2nd internal authorization server for this which is not related to google at all right

Sören Brunk · Answer

So < Fabio Grätz> so I think I had to run it through bcrypt ```pip install bcrypt python gt gt gt import bcrypt gt gt gt bcrypt hashpw b foobar bcrypt gensalt prefix=b 2a ``` The resulting hash should look something like this `b $2a$12$d3mGDJwq9F5TiQA1YYm0TOVzvEvcBX5VEw2AW0gqrn7Mvh2InuiCS ` then base64 encode it and use that as client secret in the config

Sören Brunk · Answer

I hope I remember it correctly I should have documented the steps properly

Sören Brunk · Answer

Perhaps someone from the flyte team familiar with the internal auth server can verify if this is the right way or not

Sören Brunk · Answer

Here we need the matching secret still in cleartext i e ```secrets adminOauthClientCredentials enabled true clientSecret foobar clientId flytepropeller```

Fabio Grätz · Answer

gt `b $2a$12$d3mGDJwq9F5TiQA1YYm0TOVzvEvcBX5VEw2AW0gqrn7Mvh2InuiCS ` Excluding the `b ` I suppose

Fabio Grätz · Answer

gt ```secrets gt adminOauthClientCredentials gt enabled true gt clientSecret foobar gt clientId flytepropeller``` Yes I have this slightly smiling face

Sören Brunk · Answer

Yes sorry that s from python

Fabio Grätz · Answer

Thanks a ton for looking this up < Sören Brunk>

Yee · Answer

```python c import bcrypt import base64 print base64 b64encode bcrypt hashpw mypassword encode utf 8 bcrypt gensalt 6 ```

Fabio Grätz · Answer

OMG it s working partying face

Fabio Grätz · Answer

Thanks a lot guys for the help I would have never figured this one out xD

Fabio Grätz · Answer

< Yee> Are you interested in a PR to the docs where I add this

Sören Brunk · Answer

Awesome

Fabio Grätz · Answer

I have one more question < Sören Brunk> < Yee> My understanding is that the ingress exposes routes to flyteconsole and flyteadmin so they are public in the internet To secure those we integrate e g google oauth Isn t the communication between admin scheduler and propeller purely internal to the cluster So in theory would it be fair to say that the authorization between those and admin using the built in mechanism is less important than the first one for admin and console Or are scheduler propeller available from the internet as well through a mechanism I m currently missing

Yee · Answer

sorry afk

Yee · Answer

< Fabio Grätz> yes

Yee · Answer

to the PR i mean

Yee · Answer

this was ours

Yee · Answer

```appAuth thirdPartyConfig flyteClient clientId flytectl redirectUri scopes offline all selfAuthServer staticClients flyte cli id flyte cli redirect uris grant types refresh token authorization code response types code token scopes all offline access token public true flytectl id flytectl redirect uris grant types refresh token authorization code response types code token scopes all offline access token public true flytepropeller id flytepropeller client secret JDJhJDA2JGd3N0pNUno1OXpCSzFk43DJkYlUxTHV2MGxRMlFWHNlTkczcElyU3V1TzhZai95ODJsQ2dh redirect uris grant types refresh token client credentials response types token scopes all offline access token ```

Yee · Answer

and your understanding is correct that traffic is internal to your aws account but it doesn t have to be internal to the cluster

Yee · Answer

you can run admin propeller on different clusters at lyft we ran admin on the company wide services cluster and ran a bunch of compute clusters with propeller running in each

Yee · Answer

but there s no simple secure way to say oh this is internal traffic it doesn t need auth

Yee · Answer

at least not that i m aware of

Fabio Grätz · Answer

Istio mTLS to at least not having to manage it oneself ^^

Fabio Grätz · Answer

But yes fair point I just wanted to make sure my understanding is correct that this 2nd auth mechanism isn t for any traffic publicly in the internet

Fabio Grätz · Answer

If it is traffic within our companies VPC I m a little bit less worried about it ^^

Yee · Answer

propeller scheduler no

Yee · Answer

that s all internal traffic that is correct

Fabio Grätz · Answer

So only admin to propeller traffic is traffic that could span multiple clusters

Yee · Answer

mmm

Yee · Answer

admin doesn t talk to propeller directly

Yee · Answer

in the multi cluster setup admin talks to the kube api running on the cluster

Yee · Answer

and propeller sends event information back up to admin

Fabio Grätz · Answer

Ah and this secret we encoded with `base64 b64encode bcrypt hashpw ` is used by propeller to authorize itself to admin

Yee · Answer

propeller admin talks to data catalog

Yee · Answer

yes

Yee · Answer

that is correct

Fabio Grätz · Answer

Cool

Fabio Grätz · Answer

Thanks a lot

Yee · Answer

this is for google idp for people using okta this can be different

Fabio Grätz · Answer

Wait then I still didn t understand it ^^

Fabio Grätz · Answer

Isn t the authorization between propeller and admin unrelated to google

Yee · Answer

it is but it can be related to okta

Fabio Grätz · Answer

Ok

Fabio Grätz · Answer

So for google we use the built in authorization mechanism between propeller and admin and the google idp one for accessing console and admin from the internet

Sören Brunk · Answer

It s needed because google IDP does not support oauth client credentials flow

Yee · Answer

just thought I d mention in case you write docs and it becomes relevant google idp does not come with an authorization server

Fabio Grätz · Answer

Yes I ll write docs and describe the things I needed to figure out in order to make it work after going through the docs Will tag you to correct it then ^^

Ketan (kumare3) · Answer

heart < Fabio Grätz>

Haytham Abuelfutuh · Answer

Yes < Fabio Grätz> your understanding is spot on FYI Admin will need to change a bit to accommodate non authenticated internal requests though as it currently enforces auth on all requests or no auth at all

Fabio Grätz · Answer

< Haytham Abuelfutuh> < Ketan kumare3 > Due to security policies in my new company I won t be able to deploy flyte like I now got it to work following the GCP + Auth guides where Flyte serves the google login page itself unfortunately confused Instead of the nginx ingress I will have to use the GKE native ingress controller and configure Google Identity Aware Proxy at the load balancer level so that one first hits the google login screen then the app behind it The flyte GKE docs mention google managed certificates and by modifying the ingress annotations in the helm values I think I can get this to work I already tried this yesterday but for the GKE Ingress the flytedmin backend was always unhealthy so the ingress never came up I think I might have to adapt the liveness probe of the deployment but I think i can also get that to work My question now is Assuming that when I go to and first hit Google Identity Aware Proxy log in using my company SSO and then reach the flyte UI 1 Will flyte be aware who I am from Google Idp 2 And more importantly will the authentication mechanism of flytectl pyflyte with the callback still work S I really hope the last point is not a blocker see no evil

Andrew Korzhuev · Answer

< Sören Brunk> I was having the same issue which this thread helped to resolve Shouldn t the clientSecret in `selfAuthServer staticClients` be passed through automatically Since it is available plaintext in `adminOauthClientCredentials` anyway

Ketan (kumare3) · Answer

< Fabio Grätz> sorry for the delayed response I think < Nicholas LoFaso> and team use Google IDP

Fabio Grätz · Answer

No prob +1 For now I did the following which allows me to continue with the PoC Use GKE ingress with identity aware proxy at load balancer level to comply with internal security guidelines Since the GKE ingress can t handle flyteadmins `exec command` liveness probe and the ingress never comes up I route all traffic from the ingress into an istio service mesh and via that to flyteconsole flyteadmin Make flyteadmin s gRPC endpoint available via an internal load balancer only in the VPC So engineers can register tasks only from their remote development machines or via CICD workers in the VPC But that is not too much of a limitation for us since we use remote dev machines all the time

Justin Tyberg · Answer

< Fabio Grätz> Glad you got things working I work with Nick and we have Flyte running on GCP as well This thread sums up pretty much our path with Flyte+GCP auth FWIW we re still investigating ways to streamline our auth setup

Justin Tyberg · Answer

Are you using <https cloud google com istio docs istio on gke overview|Istio on GKE> I ask because Google is deprecating that too in favor of Anthos

Fabio Grätz · Answer

Hey < Justin Tyberg> I don t use the Istio that comes with GKE when you tick the respective box but install the OOS version of istio with helm Had good experience with this Would you be interested in a short call Maybe there is knowledge we can share to streamline our setups

Justin Tyberg · Answer

< Fabio Grätz> definitely interested in syncing up and comparing notes maybe something next week

Fabio Grätz · Answer

< Justin Tyberg> yes next week sounds great Which time zone are you in

Justin Tyberg · Answer

hey < Fabio Grätz> I m in EST which is why there s a lag in my replies to you posts wink

David Espejo · Answer

So is it still true that for auth to work with Google login in a sandbox environment an encoded random secret has to be added to config so `flytepropeller` can perform auth against the K8s API instance and that this requirement is not documented yet

Ketan (kumare3) · Answer

that is correct not sure if it is not documented I thought it was documented

Haytham Abuelfutuh · Answer

This is true Unfortunately Ref for why it s removed urgently < Yee> can you share instructions for how to do that

Haytham Abuelfutuh · Answer

Oh I see Yee has shared the instructions in this thread

Ketan (kumare3) · Answer

< David Espejo> would you be open to contributing a doc addition

David Espejo · Answer

Thank you < Haytham Abuelfutuh> Yes < Ketan kumare3 > I can give it a try slightly smiling face

David Espejo · Answer

The PR is here for your review slightly smiling face <https github com flyteorg flyte pull 3062>

Fabio Grätz · Answer

< Ena Škopelja> this thread

Ketan (kumare3) · Answer

Also cc < David Espejo he him > maybe we should merge this