Hello I am trying to setup flyte on GCP GKE and although I g Flyte #ask-the-community

Hello! I am trying to setup flyte on GCP GKE and a...

Haytham Amin

07/18/2023, 5:54 PM

Hello! I am trying to setup flyte on GCP GKE and although I got all the components up and running flyteadmin is returning Not Found when I try to navigate to Dashboard. Any idea what could possibly be wrong?

David Espejo (he/him)

07/18/2023, 6:07 PM

Hi @Haytham Amin and welcome to the Flyte community. How did you setup Flyte on GKE? Regarding the error, you mean that going to

<https://your-flyte-url/console>

gives you a 404 error?

Haytham Amin

07/18/2023, 6:07 PM

correct

Haytham Amin

07/18/2023, 6:08 PM

I am using helm and terraform for setup

Haytham Amin

07/18/2023, 6:08 PM

Copy code

2023-07-17 23:00:43.994 PDT
"GA_TRACKING_ID": "",
2023-07-17 23:00:43.994 PDT
"NODE_ENV": "production",
2023-07-17 23:00:43.994 PDT
"FLYTE_NAVIGATION": ""
2023-07-17 23:00:43.994 PDT
}
2023-07-17 23:00:43.994 PDT
}
2023-07-17 23:00:43.994 PDT
Please open <http://undefined:8080/console>
2023-07-18 10:59:57.491 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:17:59:57 +0000] "GET / HTTP/1.1" 404 139 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:00:11.878 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:00:11 +0000] "GET /console HTTP/1.1" 200 1088 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:00:12.023 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:00:12 +0000] "GET /console/assets/manifest.webmanifest HTTP/1.1" 200 204 "<http://localhost:8088/console>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:00:12.417 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:00:12 +0000] "GET /console/assets/main.fb7976d2da38cf13f718.js HTTP/1.1" 200 135485 "<http://localhost:8088/console>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:00:12.542 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:00:12 +0000] "GET /console/assets/vendor.4bffaafe5f03516857b8.js HTTP/1.1" 200 1258844 "<http://localhost:8088/console>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:00:13.581 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:00:13 +0000] "GET /api/v1/projects HTTP/1.1" 404 154 "<http://localhost:8088/console>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:00:13.594 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:00:13 +0000] "GET /me HTTP/1.1" 404 141 "<http://localhost:8088/console>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:00:13.600 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:00:13 +0000] "GET /api/v1/version HTTP/1.1" 404 153 "<http://localhost:8088/console>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:00:13.963 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:00:13 +0000] "GET /console/assets/favicon.ico HTTP/1.1" 200 5430 "<http://localhost:8088/console>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:00:14.087 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:00:14 +0000] "GET /console/assets/favicon.svg HTTP/1.1" 200 2494 "<http://localhost:8088/console>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:00:20.603 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:00:20 +0000] "GET /login?redirect_url=<http://localhost:8088/console> HTTP/1.1" 404 144 "<http://localhost:8088/console>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:06:24.758 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:06:24 +0000] "GET /login HTTP/1.1" 404 144 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:06:33.050 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:06:33 +0000] "GET /console HTTP/1.1" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:06:33.186 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:06:33 +0000] "GET /console/assets/manifest.webmanifest HTTP/1.1" 200 204 "<http://localhost:8088/console>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:06:33.861 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:06:33 +0000] "GET /api/v1/projects HTTP/1.1" 404 154 "<http://localhost:8088/console>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"
2023-07-18 11:06:34.378 PDT
::ffff:127.0.0.1 - - [18/Jul/2023:18:06:34 +0000] "GET /api/v1/version HTTP/1.1" 404 153 "<http://localhost:8088/console>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15"

David Espejo (he/him)

07/18/2023, 6:15 PM

ok, what are you using for Ingress?

David Espejo (he/him)

07/18/2023, 6:16 PM

Or are you port-forwarding?

Haytham Amin

07/18/2023, 6:16 PM

right now port forwarding as ingress is not working

Haytham Amin

07/18/2023, 6:17 PM

flyteconsole returns the banner with 404 not found as well

David Espejo (he/him)

07/18/2023, 6:36 PM

what's the status of the pods?

Haytham Amin

07/18/2023, 7:12 PM

All up and running

Geert

07/18/2023, 7:52 PM

Hey, I think Im hitting this as well running on local K3d/Helm deployment, all Pods up and running:

David Espejo (he/him)

07/18/2023, 7:53 PM

anything interesting in the Pod's logs?

Geert

07/18/2023, 8:10 PM

Not much honestly, all looks pretty good. Tried reverting to flyteconsole-release:v1.7.0 but that also has issues weirdly. Haven't changed much since it ran w/o issues, perhaps something external changed?

Geert

07/18/2023, 8:13 PM

Only thing that looks a bit off is the

undefined

Please open <http://undefined:8080/console>

Geert

07/18/2023, 8:14 PM

Though I can access (after port-forwarding) localhost:8080/console, so Im guessing that shouldnt be an issue

Haytham Amin

07/18/2023, 8:15 PM

Can confirm 1.7.0 and 1.8.0 same behavior

Geert

07/18/2023, 8:20 PM

Im guessing this is related: https://github.com/flyteorg/flyte/issues/416

Geert

07/18/2023, 8:27 PM

Can confirm it works after: • port-forwarding

flyteadmin

to localhost:8089 • updating configmap

flyte-console-config

to add env variable ADMIN_API_URL: <http://localhost:8089>
• restart flyteconsole pod

David Espejo (he/him)

07/18/2023, 8:36 PM

thanks Geert!, that makes sense for sandbox nevertheless @Haytham Amin in your case, as you're not using sandbox, I don't think it applies

Haytham Amin

07/18/2023, 8:40 PM

I can try to add the env var

Haytham Amin

07/18/2023, 8:40 PM

In the logs I see url undefined:8088

Haytham Amin

07/18/2023, 9:25 PM

Copy code

kubectl -n flyte port-forward svc/flyteconsole 8088:80  
Forwarding from 127.0.0.1:8088 -> 8080
Forwarding from [::1]:8088 -> 8080
Handling connection for 8088
Handling connection for 8088
Handling connection for 8088

Haytham Amin

07/18/2023, 9:26 PM

image.png

Haytham Amin

07/18/2023, 9:28 PM

in values.yaml I have the following in config map:

Copy code

# -- Authentication configuration
  auth:
    authorizedUris:
      # This should point at your public http Uri.
      - "https://{{ .Values.userSettings.hostName }}"
      # This will be used by internal services in the same namespace as flyteadmin
      - <http://flyteadmin:80>
      # This will be used by internal services in the same cluster but different namespaces
      - <http://flyteadmin.flyte.svc.cluster.local:80>

Haytham Amin

07/18/2023, 9:54 PM

image.png

David Espejo (he/him)

07/19/2023, 11:11 AM

@Haytham Amin what else you have in

configmap

? so you plan on using auth? I don't think a problem in auth can lead to a 404 in console, so we can leave auth disabled for now if possible

Haytham Amin

07/19/2023, 7:33 PM

@David Espejo (he/him)

Copy code

#
# CONFIGMAPS
#

configmap:
  adminServer:
    server:
      httpPort: 8088
      grpcPort: 8089
      security:
        secure: false
        useAuth: false
        allowCors: true
        allowedOrigins:
          # Accepting all domains for Sandbox installation
          - "*"
        allowedHeaders:
          - "Content-Type"

  # -- Authentication configuration
  # auth:
  #   authorizedUris:
  #     # This should point at your public http Uri.
  #     - "https://{{ .Values.userSettings.hostName }}"
  #     # This will be used by internal services in the same namespace as flyteadmin
  #     - <http://flyteadmin:80>
  #     # This will be used by internal services in the same cluster but different namespaces
  #     - <http://flyteadmin.flyte.svc.cluster.local:80>        

  task_resource_defaults:
    task_resources:
      defaults:
        cpu: 1
        memory: 500Mi
        storage: 500Mi
      limits:
        storage: 2000Mi

  # Adds the remoteData config setting
  remoteData:
    remoteData:
      region:
      scheme: "gcs"
      signedUrls:
        durationMinutes: 3

  # Adds the namespace mapping to default to only domain name instead of project-domain in case of GCP
  namespace_config:
    namespace_mapping:
      template: "{{ domain }}"

  core:
    propeller:
      rawoutput-prefix: "gs://{{ .Values.userSettings.bucketName }}/"
      workers: 40
      gc-interval: 12h
      max-workflow-retries: 50
      kube-client-config:
        qps: 100
        burst: 25
        timeout: 30s
      queue:
        sub-queue:
          type: bucket
          rate: 100
          capacity: 1000

  enabled_plugins:
    # -- Tasks specific configuration [structure](<https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig>)
    tasks:
      # -- Plugins configuration, [structure](<https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig>)
      task-plugins:
        # -- [Enabled Plugins](<https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/config#Config>). Enable sagemaker*, athena if you install the backend
        # plugins
        enabled-plugins:
          - container
          - sidecar
          - k8s-array
          - agent-service
        default-for-task-types:
          container: container
          sidecar: sidecar
          container_array: k8s-array
          bigquery_query_job_task: agent-service


  # -- Section that configures how the Task logs are displayed on the UI. This has to be changed based on your actual logging provider.
  # Refer to [structure](<https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/logs#LogConfig>) to understand how to configure various
  # logging engines
  task_logs:
    plugins:
      logs:
        kubernetes-enabled: false
        # Enable GCP stackdriver integration for log display
        stackdriver-enabled: true
        stackdriver-logresourcename: k8s_container
      k8s-array:
        logs:
          config:
            stackdriver-enabled: true
            stackdriver-logresourcename: k8s_container

Haytham Amin

07/19/2023, 7:34 PM

@David Espejo (he/him) I think the docs on GCP deployment needs some work.

47 Views

Open in Slack

Previous Next