Hi All, I am trying to POC in the restricted env u...
# ask-the-community
Hi All, I am trying to POC in the restricted env using single cluster deployment. In our environment, We cannot give permissions to create namespace and service accounts to flyte's system level service account. We can create namespace and service accounts using pipelines. Is there a way to configure flyte to use existing namespace and service accounts for flyte project. Right now it is trying to create namespaces and service accounts even if they exist?
Yes, Flyte can be restricted to one or per domain namespace (recommended)
Cc @David Espejo (he/him)
@peeyush jain you can prevent Flyte from creating service accounts, you can also as Ketan indicated, make Flyte use a single namespace but I'm not sure if what you need is to prevent Flyte from creating any namespace by default?
@David Espejo (he/him) Thanks for response. How can i prevent flyte from creating service accounts? what config and value do i need to achieve that? Regarding the namespace i can create namespaces using pipelines. Will it possible to use pre-created namespaces?
@David Espejo (he/him) Flyte was able to use existing namespaces. After commenting the block for service account (002_service_Account.yaml) from clusterResourceTemplates, i was able to move forward. Not sure if this is best way? will it now use the system service account for each project namespace we created manually?
@peeyush jain you can disable SA creation by setting the following key to `false`: https://github.com/flyteorg/flyte/blob/a2a63eb893bdff3c2d1a01242d0d25b73f287b1f/charts/flyte-binary/values.yaml#L371-L373 Or this key for flyteadmin and flytepropeller, in cas you're using the flyte-core Helm chart: https://github.com/flyteorg/flyte/blob/a2a63eb893bdff3c2d1a01242d0d25b73f287b1f/charts/flyte-core/values-eks.yaml#L19
will it now use the system service account for each project namespace we created manually?
still a `default`SA will be available on each namespace that will be used by the Pods bootstrapped by Flyte. The
referenced on the charts it's typically then annotated to be used in conjunction with the IAM module of the cloud provider. Where are you running Flyte?
Thanks. I am deploying on EKS in aws. I am using eks-starter.yaml for helm chart deployment. I am also looking for any guide to use AWS secret manager instead of K8s for secrets pluglin. Any pointers will be helpful.
I saw secretManagerType config for propeller. I tried to set it in core config (000-core.yaml) in the configmap.
As below. But it din't work. It is still tyring to create secret in K8s propeller: create-flyteworkflow-crd: true secretManagerType: AWS
no guide yet but a workaround proposed here: https://github.com/flyteorg/flyte/issues/4336
Thanks. For time being i got admin to open up permissions on cluster roles. I was able to complete the deployment. Now when i try to flytectl it throws error flytectl -p flytesnacks -d development get tasks Roc error: code = unavailable desc = connection error: desc = \”transport: authentication handshake failed: context deadline exceeded” Using below admin conf endpoint: dns:///ml-platform-poc.xxxxxxxx.xxx authType: pkce insecure: false insecureSkipVerify: true Pyflyte run —remote example.py hello_wrold_wf fails with Debug string UNKNOWN: Error received from peer {gprc_message:”Received http2 header with status 464, sprc_status:2 created_time”2024-03-01T112226.47676767+00:00”}
@David Espejo (he/him) I believe the above error has to do with ALB configuration. We are using NGINX ingress controller. any inputs on how the ALB routes should be configured for gprc service?
@David Espejo (he/him) I was able to configure the external ALB to run with single cluster deployment. In our env, with external ALB the ssl needs to be terminated at pod. If i add config map /seceret with self signed certificate and key. Can i use ssl options in ServerSecurityConfig to run flyte binary pods with ssl?
@peeyush jain the ALB config on these lines is typically enough to allow you to terminate at the load balancer using an ACM-managed certificate
Copy code
  enabled: true
  secretName: flyte-secret-tls
also adding this block, and assuming you have a K8s secret with your certificate on it, you could still terminate it on the LB but using TLS
Thanks @David Espejo (he/him) In our env, we are not allowed to alb as native ingress controller. We create alb and targetgroups and bindings externally via tfe. Nginx is supported as native controller but it does not support grpc protocol in our environment. With nginx flyte console works fine but when using flytectl i get error "recieved http2 header, status 464". Hence i switched to external alb for ingress which allows grpc but ssl has to terminate on pod. I am using single cluster deployment eks-starter.yaml. Will need your inputs on how to configure ssl for flyte binary pod.
I've seen situations with other Ingress controllers that switched to NGINX because it supports gRPC. Using these annotations should work:
Copy code
ingressClassName: nginx
    ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/app-root: /console
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
as per terminating SSL on the Pod itself, I'd explore this NGINX feature plus the Flyte-side config you already saw Doing SSL passthrough with ALB is an area where I lack experience tbh
Thanks David. I will create a ssl proxy service and see if it works