I'm having issues accessing the grpc endpoint behind an AWS ELB. We have flyte installed by followi...
r

Rob Rati

over 2 years ago
I'm having issues accessing the grpc endpoint behind an AWS ELB. We have flyte installed by following this: https://docs.flyte.org/en/latest/deployment/deployment/cloud_production.html And we have the http endpoint exposed via alb w/o issue. We use the aws ingress controller to create the ALBs, so I've modified the grpc ingress definition to add the following annotations: alb.ingress.kubernetes.io/backend-protocol: HTTP alb.ingress.kubernetes.io/backend-protocol-version: GRPC alb.ingress.kubernetes.io/certificate-arn: <redacted> alb.ingress.kubernetes.io/healthcheck-path: /grpc.health.v1.Health alb.ingress.kubernetes.io/healthcheck-protocol: HTTP alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' alb.ingress.kubernetes.io/scheme: internal alb.ingress.kubernetes.io/security-groups: EKS-ALB-sg alb.ingress.kubernetes.io/target-type: ip And the spec with: ingressClassName: alb rules: - host: <http://flyte-web-east2-grpc.api-platform-dev.aws-nonprod.fmrcloud.com|<>hostname> But whenever I try to use grpcurl to access the endpoint to list/describe I always get: $ grpcurl <hostname>:443 list Failed to list services: server does not support the reflection API $ grpcurl <hostname>:443 describe /flyteidl.service.IdentityService Failed to resolve symbol "/flyteidl.service.IdentityService": server does not support the reflection API When I try to use grpc_cli ls <hostname>:443 I get an error: Trying to connect an http1.x server I can describe the grpc endpoints when I port-forward the grpc service, so the server is working as expected. Has anyone else tried to do this?
Having trouble running the *`k8s_spark.<http://dataframe_passing.my|dataframe_passing.my>_smart_stru...
c

Chase Grisham

over 3 years ago
Having trouble running the
k8s_spark.<http://dataframe_passing.my|dataframe_passing.my>_smart_structured_dataset
example. I've been able to run the other spark example
pyspark_pi
. I've set up the K8s Operator, built the Docker Image based on the Dockerfile in the
cookbook/integrations/kubernetes/k8s
folder. Getting this error about s3 :
[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):

      File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py", line 527, in dispatch_execute
        raise TypeError(

Message:

    Failed to convert return value for var o0 for function k8s_spark.dataframe_passing.create_spark_df with error <class 'py4j.protocol.Py4JJavaError'>: An error occurred while calling o41.parquet.
: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
	at org.apache.hadoop.fs.FileSystem.get(FiInternal(DataFrameWriter.scala:355)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
	at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:781)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:748)


SYSTEM ERROR! Contact platform administrators.
I need help in configuring flyte with Keycloak I have setup 2 clients, flytectl(access type public)...
s

Sujith Samuel

about 3 years ago
I need help in configuring flyte with Keycloak I have setup 2 clients, flytectl(access type public) and flytepropeller(access type confidential with client Id and secret). I am following the keycloak section in the https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html# My relevant values in the helm values file are as below secrets: adminOauthClientCredentials: # -- If enabled is true, helm will create and manage
flyte-secret-auth
and populate it with
clientSecret
. # If enabled is false, it's up to the user to create
flyte-secret-auth
as described in # https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#oauth2-authorization-server enabled: true clientSecret: "CO2n3hovN0J78FqxxOVtjOtkHH5fPL9C" clientId: "flytepropeller" -- FlyteAdmin server configuration adminServer: # Refer to the server config. server: httpPort: 8088 grpcPort: 8089 security: # -- Controls whether to serve requests over SSL/TLS. secure: false # -- Controls whether to enforce authentication. Follow the guide in https://docs.flyte.org/ on how to setup authentication. useAuth: true allowCors: true allowedOrigins: # Accepting all domains for Sandbox installation - "*" allowedHeaders: - "Content-Type" - "flyte-authorization" # Refer to the full structure for documentation. flyteadmin: roleNameKey: "iam.amazonaws.com/role" profilerPort: 10254 metricsScope: "flyte:" metadataStoragePrefix: - "metadata" - "admin" eventVersion: 2 testing: host: http://flyteadmin # -- Authentication configuration auth: authorizedUris: # This should point at your public http Uri. - https://flytedeployment url # This will be used by internal services in the same namespace as flyteadmin - http://flyteadmin:80 # This will be used by internal services in the same cluster but different namespaces - http://flyteadmin.nmlp.svc.cluster.local:80
# Controls app authentication config
  appAuth:
    thirdPartyConfig:
      flyteClient:
        clientId: flytectl
        redirectUri: https://<flyte deployment url>/callback
        scopes:
          - offline
          - all

  # Controls user authentication
  userAuth:
    openId:
      baseUrl: https://<keycloak production realm>/realms/nsdmlp
      scopes:
        - profile
        - openid
      clientId: flytepropeller
I have also edited the flyte-admin-secrets to have the correct client secret apiVersion: v1 data: claim_symmetric_key: cWlBYzlYWHdLN3lnaksrWUJGdStFUlRYK0RDdlk4SjVjZFJtaXBTcDBhdw== cookie_block_key: ejZPdkhrZ1crWXdib21JZHdVZ05IOGJESVp0OE5KWnNZT285KzIyRVM1dw== cookie_hash_key: Q093TUY2RTdOMW5MeFZ4Rnk1dGNzZGN5NU5aeTNWN2JTMXRPTjBLUGhQQ2JWZ3hGby9XQkVRdi84Yjk4ZEIyeEV3Zm5KYURDVzFkSjBuSGZrbS8zYVE= oidc_client_secret: CO2n3hovN0J78FqxxOVtjOtkHH5fPL9C With all the above setting, when i try to login to the flyte console via the ingress, it gives me the below message in the flyteadmin pod logs {"json":{},"level":"error","msg":"Failed to retrieve tokens from request, redirecting to login handler. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_idt], caused by: http: named cookie not present","ts":"2022-08-04T183411Z"} {"json":{},"level":"error","msg":"Error when exchanging code oauth2: cannot fetch token: 401 Unauthorized\nResponse: {"error":"unauthorized_client","error_description":"Invalid client secret"}","ts":"2022-08-04T183412Z"} Logs from 8/5/2022, 120239 AM And the chrome errors out with 403 access denied page. I have tried secret and id with and without quotes(just in case if helm was acting crazy) but it still gives the same error. Please assist. Thanks, Sujith