Hi I m just getting started with a test Flyte deployment on

Question

Hi I m just getting started with a test Flyte deployment on GCP with the goal of running pyflyte with the remote flag to start I m using the automated GCP setup with Opta Is it possible to deploy and run Flyte on GCP without providing a registered domain name I d like to try out the deployment but do not wish to register a domain at this stage unless absolutely necessary I was able to deploy Flyte by commenting out the domain fields and disabling ingress in the config But was wondering if at that stage I can port forward something to get everything needed to run pyflyte run remote

Ketan (kumare3) · Answer

Cc < Prafulla Mahindrakar> < jeev> < Sören Brunk>

Prafulla Mahindrakar · Answer

yes that should be possible if you just port forward flyteadmin and use that endpoint in the pyflyte config yaml eg ```admin For GRPC endpoints you might want to use dns flyte myexample com endpoint dns port forwarded uri authType Pkce insecure true```

jeev · Answer

will break forwarding to flyteconsole probably in case he s trying to monitor our local sandbox uses a envoy proxy for this exact purpose

Prafulla Mahindrakar · Answer

yeah thats true but we can monitor from flytectl aswell Jeev do you have the steps with envoy proxy documented somewhere which you can share

jeev · Answer

this is the config we use ```apiVersion apps v1 kind Deployment metadata name flyte proxy labels app flyte proxy spec selector matchLabels app flyte proxy template metadata labels app flyte proxy spec containers name proxy image envoyproxy envoy v1 21 1 args envoy c etc envoy config yaml ports name http containerPort 8000 volumeMounts name config volume mountPath etc envoy volumes name config volume configMap name flyte proxy config```

jeev · Answer

will probably want to drop the unnecessary stuff like minio k8s dashboard etc

jeev · Answer

and you can just `kubectl port forward deploy flyte proxy 8000`

jeev · Answer

perhaps worth PR ing this < Prafulla Mahindrakar> we don t use this on our prod deploys but its nice for sandboxes

jeev · Answer

i think the current flyte sandbox just uses contour ingress controller instead though

jeev · Answer

but that might not be ideal to deploy to the prod cluster

Prafulla Mahindrakar · Answer

I think this can be additionally be done after opta deployment and not use it for prod deployments This can be part of the docs once we get it working with opta deployed flyte For sandbox we already have contour and that should be sufficient for trials locally

Tom Szumowski · Answer

< Prafulla Mahindrakar> re gt yes that should be possible if you just port forward flyteadmin and use that endpoint in the pyflyte config yaml I tried something similar I believe and got some RPC errors First I did ```kubectl n flyte port forward service flyteadmin 30081 81``` Then I set my ` flyte config yaml` to ```admin For GRPC endpoints you might want to use dns flyte myexample com endpoint dns localhost 30081 authType Pkce insecure true logger show source true level 0``` Then I ran ```$ pyflyte run remote core flyte basics basic workflow py my wf a 5 b hello``` The error I got is in this snippet ``` debug error string = created 1658972857 863942000 description Error received from peer ipv6 1 30081 file src core lib surface call cc file line 904 grpc message failed to create a signed url Error unable to sign bytes googleapi Error 403 The caller does not have permission grpc status 2 ``` I m brand new here so its very possible I missed a setup step somewhere

Tom Szumowski · Answer

< jeev> re gt will break forwarding to flyteconsole probably in case he s trying to monitor our local sandbox uses a envoy proxy for this exact purpose 1 I m still learning Can you describe why it breaks forwarding to flyteconsole Naively I did try to additionally forward flyteconsole along with the above using ```kubectl n flyte port forward service flyteconsole 30080 80``` I saw the console when I navigated to `localhost 30080 console` but there was an error displayed I m curious why Thank you 2 Silly question But once the everything is deployed via Opta how do you apply that envoy k8s config and layer in the envoy config yaml

Prafulla Mahindrakar · Answer

Can you check if the service account for flyteadmin and corresponding gcp linked service account has permissions to create signed url

Tom Szumowski · Answer

< Prafulla Mahindrakar> sure I destroyed the env last night EDT before going to sleep I ll re deploy via opta right now and check that

Tom Szumowski · Answer

< Prafulla Mahindrakar> Deployed Hmm Seems like the service accounts were never created after running `opta apply c flyte yaml` Tried re running and no resources changed The output says ```adminflyteaccount service account email = gsa flyteadmin lt GCP PROJECT gt adminflyteaccount service account id = gsa flyteadmin bucket id = lt NAME gt service flyte bucket name = lt NAME gt service flyte datacatalogaccount service account email = gsa datacatalog lt GCP PROJECT gt datacatalogaccount service account id = gsa datacatalog flytedevelopmentaccount service account email = gsa development lt GCP PROJECT gt flytedevelopmentaccount service account id = gsa development flyteproductionaccount service account email = gsa production lt GCP PROJECT gt flyteproductionaccount service account id = gsa production flytepropelleraccount service account email = gsa flytepropeller lt GCP PROJECT gt flytepropelleraccount service account id = gsa flytepropeller flytescheduleraccount service account email = gsa flytescheduler lt GCP PROJECT gt flytescheduleraccount service account id = gsa flytescheduler flytestagingaccount service account email = gsa staging lt GCP PROJECT gt flytestagingaccount service account id = gsa staging ``` However I don t see any of the `gsa ` service accounts in my project IAM settings I only see one new one ```opta lt NAME gt ep63 lt GCP PROJECT gt ``` thinking face

Prafulla Mahindrakar · Answer

hmm thats strange did the opta apply complete thats seems odd since it should have created the gsa s

Prafulla Mahindrakar · Answer

adding opta folks on this thread < JD Palomino>

Tom Szumowski · Answer

It said it completed The above is from the end of the run where it reports out everything

Tom Szumowski · Answer

Here s the output from the last step the helm chart run

Tom Szumowski · Answer

Oh I see ```Warnings Applied changes may be incomplete To see the full warning notes run Terraform without compact warnings ``` That may be related Is there a way to re run the Opta or terraform directly to give full warnings

Prafulla Mahindrakar · Answer

It seems it only applied the helm chart and not recreate any env stuff can you destroy the env and retry

Tom Szumowski · Answer

env yaml flyte yaml or both

Prafulla Mahindrakar · Answer

both

Tom Szumowski · Answer

Ok will do

Prafulla Mahindrakar · Answer

Also please capture the logs from opta runs so that the opta team can also have a look in case of issues

Tom Szumowski · Answer

Here are the logs from me re deploying env yaml and flyte yaml I had to re run once or twice due to internet connection dropouts Same outcome No `gsa ` service accounts in my GCP IAM page like before The output is using just `opta apply c flyte yaml` If there is a way to get more verbose logs I can re run

Tom Szumowski · Answer

Ah I do see a ` detailed plan` option I ll destroy and re apply flyte yaml with that option to print more out

Tom Szumowski · Answer

point up update face palm I see the service accounts now I either did not properly refresh the page or checked at the wrong time Disregard the opta issues above I ll continue on checking service account permissions from <https flyte org slack com archives C01P3B761A6 p1659004801277439 thread ts=1658974292 802369 amp cid=C01P3B761A6|here> when I have a moment to poke Apologies for the confusion

Prafulla Mahindrakar · Answer

ahh i see np check the bucket write permissions Check if you have this permission iam serviceAccounts signBlob It also mentioned in the manual deployment doc here <https docs flyte org en latest deployment gcp manual html permissions>

Tom Szumowski · Answer

Interesting It doesn t seem any of those Roles were created or applied The account does have access to the buckets though

Tom Szumowski · Answer

I m currently trying to see where `iam serviceAccounts signBlob` permission is provided to the flyteadmin service in the Opta configuration

Prafulla Mahindrakar · Answer

that permission might be missing from opta probably since signed url feature came in later Do you mind trying to add a role that contains permission and add ``` serviceAccount gsa flyteadmin urbn data science iam gserviceaccount com``` as a member

Tom Szumowski · Answer

added I m able to successfully submit a run ```$ pyflyte run remote core flyte basics basic workflow py my wf a 5 b hello Go to <http localhost 30081 console projects flytesnacks domains development executions f75075fdeda774b358b4> to see execution in the console ```

Tom Szumowski · Answer

Naturally `localhost 30081` doesn t have the console UI for reasons discussed above still curious why port forwarding 30080 doesn t work that s for another time wink But I can access the logs using flightctl

Tom Szumowski · Answer

Tom Szumowski · Answer

Still learning the tools so I just used kubectl instead in order to see the pod error Looks like a `storage objects get` permissions issue now

Prafulla Mahindrakar · Answer

ohh can you try giving all the roles mentioned in the manual gcp doc

Tom Szumowski · Answer

I m wondering if that s because of perhaps missing workload identity step in that doc

Prafulla Mahindrakar · Answer

workload identity is enabled by opta automatically for gcp so i doubt that could be an issue

Tom Szumowski · Answer

Ok I was curious because I didn t see the mentioned role `roles iam workloadIdentityUser` in my project anywhere

Tom Szumowski · Answer

trying out the roles taking a moment since creating the roles from scratch

Prafulla Mahindrakar · Answer

can you check what service account did the pod use and also describe that service account With workfload identity you should see an annotation on the service account which maps the gsa and that should have right roles

Prafulla Mahindrakar · Answer

for your execution i think it should show up annotation for this ```serviceAccount gsa development urbn data science iam gserviceaccount com```

Tom Szumowski · Answer

Would you be able to describe how to get that info

Tom Szumowski · Answer

I did a `k n development describe pods f75075fdeda774b358b4 n0 0` and got this output But didn t see a service account there

Prafulla Mahindrakar · Answer

kubectl get pod n flytesnacks development lt name of exec pod gt o yaml |grep service And then use the name of serviceAccount to do a get kubectl get sa n flytesnacks development lt serviceAccount name gt o yaml

Tom Szumowski · Answer

Oh my mistake I saw only a `default` service account on the k8s cluster and assumed I should have something different

Tom Szumowski · Answer

Output from the pod describe ``` serviceAccount default serviceAccountName default```

Tom Szumowski · Answer

Output from service account describe ```$ k n development get sa default o yaml apiVersion v1 kind ServiceAccount metadata annotations creationTimestamp 2022 07 28T13 38 02Z name default namespace development resourceVersion 31160 uid 0bda0bec 49e4 4b32 991d 7fd706315c77 secrets name default token zfswb```

Prafulla Mahindrakar · Answer

ok so the mapped gsa annotation looks correct ``` ``` This particular gsa needs to have those storage roles

Tom Szumowski · Answer

Just finished adding the roles and applying them to the SAs Will try re running pyflyte

Tom Szumowski · Answer

Bummer No luck with all those roles and permissions applied manually Same error as above

Tom Szumowski · Answer

I ll need to park it here to take care of some other things Hopefully will get back to this experiment later today slightly smiling face

Ketan (kumare3) · Answer

Would It be easier to get on a call

Tom Szumowski · Answer

Yeah that may be easier Unfortunately I have meetings most of rest of today EDT so I can circle back another time to coordinate +1

Prafulla Mahindrakar · Answer

there something wrong with workload identity setup probably You can try this tutorial from gcp which lets you create a wi test pod and call into the metadataserver to check if its using right identity <https cloud google com kubernetes engine docs how to workload identity> You can try to use the default service account in flytesnacks development ns later if you are able to test out the same in another test ns

Prafulla Mahindrakar · Answer

Also since we got past the signed url permissions on admin with your role changes might be worth checking what s different with the user pod service account and there roles They should ideally work the same way

Tom Szumowski · Answer

I had a moment to poke this afternoon Thanks for the link < Prafulla Mahindrakar> Here s the current state of affairs on my end 1 large green circle Confirmed tutorial workload identity process worked I walked through that tutorial and confirmed that in that I can see the WI and GCP service accounts annotated ```$ kubectl exec it pod workload identity test namespace test wi bin bash root workload identity test curl H Metadata Flavor Google default ``` 2 large green circle Confirmed I can get read the bucket with a test pod inside the `development` flyte namespace Using this test spec I was able to read the Flyte bucket in the container by executing `gsutil ls ` in the pod Spec flight test yaml ```apiVersion v1 kind Pod metadata name flyte manual test namespace development spec containers image image google cloud sdk slim name flyte manual test command sleep infinity resources limits cpu 500m memory 500Mi requests cpu 500m memory 500Mi``` Output ```root workload identity test gsutil ls ``` This is using ` ` as the GCP mapped SA 3 red circle Unable to run pyflyte or even same spec with flytekit image pyflyte runs still result in permissions errors like above Another interesting note if I swap the image in the above spec from `google cloud sdk slim` to `` I get the same permission errors Output when using ``` in flyte test yaml ```root flight test ts temp ~ gsutil ls ServiceException 401 Anonymous caller does not have storage objects list access to the Google Cloud Storage bucket ``` same error I confirmed the service account in that container is the same as above by hitting the endpoint shown in 1 from the tutorial but using python requests since curl isn t in that flytekit image ```root flyte manual test ~ python3 gt gt gt import requests gt gt gt r = requests get headers= Metadata Flavor Google gt gt gt print r content decode default ``` It also uses ` ` I also pulled the spec from the pyflyte execution and attempted a manual `gsutil` on the running pod entering with a sleep and got the same error

Tom Szumowski · Answer

Will park here for now again since didn t have too much time to play again slightly smiling face

Sören Brunk · Answer

< Tom Szumowski> this might be related to standalone `gsutil` not being able to authenticate without additional config compared to google cloud sdk installed gsutil I ran into this a while ago and we actually had a thread here I can t find anymore probably due to the Slack history limit

Sören Brunk · Answer

Looks like the flytekit image installs standalone <https github com flyteorg flytekit blob 490597a2a2b159388734cfbafa499a8749d6cc51 Dockerfile py3 8 L10|gsutil via pip>

Sören Brunk · Answer

Let me look for a Dockerfile with that additional configuration needed for gsutil

Sören Brunk · Answer

Ok found it This is what we do The boto config is the important part ```RUN curl | tar xfz C opt amp amp ln s opt gsutil gsutil bin gsutil RUN echo GoogleCompute nservice account = default gt etc boto cfg Required for gsutil to work with workload identity``` I guess it should also work with standalone gsutil installed via pip so perhaps try to derive an image from `` with the second line added to check if that makes any difference

Tom Szumowski · Answer

< Sören Brunk> interesting Thank you for the advice Will try it out when back on the laptop So in your case do you have a custom GCP deployment of Flyte that doesn t use or extends from the flytekit image

Ketan (kumare3) · Answer

< Sören Brunk> did you not find it better to use `flytekitplugins data fsspec` and then install GCS for fsspec

Ketan (kumare3) · Answer

so < Tom Szumowski> the storage layer is configurable

Sören Brunk · Answer

< Tom Szumowski> we have something similar to the manual deployment in the docs but with terraform for better integration into our existing infra And yes we use a custom flytekit image as well but mostly for historical reasons the official one didn t exist back then

Sören Brunk · Answer

< Ketan kumare3 > I think we had an issue with fsspec back then which caused us to use gsutil again Not sure if it s still there I have to try again with current flytekit

Tom Szumowski · Answer

< Sören Brunk> success tada I used this Dockerfile ```FROM Required for gsutil to work with workload identity RUN echo GoogleCompute nservice account = default gt etc boto cfg``` Pushed it as image name `````` Then ran pyflyte ```pyflyte run image remote core flyte basics basic workflow py my wf a 5 b hello``` And got a successful run on the GKE cluster ```$ flytectl get execution ffe50f73fa4564737bf6 p flytesnacks d development | NAME | LAUNCH PLAN NAME | VERSION | TYPE | PHASE | SCHEDULED TIME | STARTED | ELAPSED TIME | ABORT DATA TRUNC | ERROR DATA TRUNC | | ffe50f73fa4564737bf6 | core flyte basics basic workflow my wf | 00jRSrIIdnwryVi5J7STWw== | LAUNCH PLAN | SUCCEEDED | | 2022 07 28T23 52 37 007821370Z | 78 295864832s | | | ``` Thank you everyone in this thread for the fantastic support I think with this tweak that concludes the investigation for the original goal i e getting flyte to run on GCP without a domain The only open item on my end is to try out that envoy config < jeev> provided in order to see the GUI perhaps tomorrow

Tom Szumowski · Answer

To summarize some issues discovered 1 Opta install did not appear to install the roles permissions and workflow identity permissions I had to follow manual instructions for that to apply 2 The line ` GoogleCompute nservice account = default ` is required in ` etc boto cfg` for the flytekit docker image to work Otherwise the pod dies with a bucket permission error though I am guessing this since this is cloud specific it should be handled somewhere else outside the Dockerfile maybe Do you all suggest I report these as GitHub issues Or are they known already and a report not required

Ketan (kumare3) · Answer

Yes please

Ketan (kumare3) · Answer

Thank you Tom

Tom Szumowski · Answer

No problem This was fun

jeev · Answer

< Tom Szumowski> let me know if you cant get envoy working it should work oob i think slightly smiling face

Tom Szumowski · Answer

< jeev> success Worked oob Just needed to create the configmap then deploy to flyte namespace and I was able to port forward 8000 Thank you

Tom Szumowski · Answer

< jeev> hmm I may have celebrated too soon I am able to access the console but unable to get pyflyte to connect to flyteadmin and get a workflow submitted I port forward envoy with ```kubectl n flyte port forward deployment flyte proxy 30080 8000``` I then set my `~ flyte config yaml` to ```admin For GRPC endpoints you might want to use dns flyte myexample com endpoint dns localhost 30080 ``` And when I run I get the error ``` Non auth RPC error lt InactiveRpcError of RPC that terminated with n tstatus = StatusCode UNIMPLEMENTED ``` full trace attached I can still get it to work if I separately port forward flyteadmin with ```kubectl n flyte port forward service flyteadmin 30081 81``` and set config to `30081` When you use this are you able to access flyteadmin and the console with one port forward

jeev · Answer

yes a single port forward should suffice

jeev · Answer

it s possible that the config is missing a necessary endpoint

jeev · Answer

if you are able to browse your projects in flyteconsole you are connected to flyteadmin as well

jeev · Answer

let s just figure out the grpc endpoint and we should be good

Tom Szumowski · Answer

Thanks My guess is in me hacking away at the Opto config a bit I may have inadvertently removed an endpoint I ll poke around at some point but trust the envoy config is good to go and in the meantime I can always keep that second port forwarded to test other things Thanks again

jeev · Answer

the envoy config might be missing an endpoint that i just haven t used yet might be entirely possible

jeev · Answer

the exhaustive list is here <https github com flyteorg flyte blob master charts flyte core templates common ingress yaml|https github com flyteorg flyte blob master charts flyte core templates common ingress yaml>

jeev · Answer

alternatively if you already created an ingress you can just refer to that