Thread
#ask-the-community
    e

    Eugene Cha

    9 months ago
    Hi all. I originally had flyte working, but had to delete the cluster and reinstall. I passed the deployment stage and set it using this console (https://birdejo.live). Unfortunately the console doesnt seem to work. Any ideas on how to diagnose what the issue is?
    Ketan (kumare3)

    Ketan (kumare3)

    9 months ago
    What do you mean not working?
    Can you check the pods in the k8s cluster- Flyte namespace
    cc @Prafulla Mahindrakar / @Yuvraj incase @Eugene Cha happens to ask again
    e

    Eugene Cha

    9 months ago
    On the admin pods I get a no update handler registered as well as on the webhook pods
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [flyteadmin] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [scheduler] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [remotedata] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [cluster_resources] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [qualityofservice] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [registration] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [notifications] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [domains] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [task_resources] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [secrets] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [queues] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [namespace_mapping] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [task_type_whitelist] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [server] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.catalogcache] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.k8s] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
    Ketan (kumare3)

    Ketan (kumare3)

    9 months ago
    Thatโ€™s ok, but the pods are up and not crashing right?
    p

    Prafulla Mahindrakar

    9 months ago
    That shouldn't cause the issue you are seeing . Can you paste details if all the flyte pods are runnings and also
    Also can you check the ingress details aswell using kubectl get ingress -n flyte
    e

    Eugene Cha

    9 months ago
    pods are up and not crashing
    ingress seems to be working
    basically can't connect to the console
    p

    Prafulla Mahindrakar

    9 months ago
    Can you also dump the ingress details. kubectl get ingress -n flyte flyte-core -o yaml kubectl get ingress -n flyte flyte-core-grpc -o yaml
    e

    Eugene Cha

    9 months ago
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      annotations:
        <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>: '{"Type": "redirect", "RedirectConfig":
          { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
        <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: <CERTIFICATE_ARN>
        <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
        <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTP": 80}, {"HTTPS":443}]'
        <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
        <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
        <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
        <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: service-flyte-helmchart
        <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
        <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
      creationTimestamp: "2021-12-27T01:48:25Z"
      generation: 1
      labels:
        <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
      name: flyte-core
      namespace: flyte
      resourceVersion: "131147"
      selfLink: /apis/extensions/v1beta1/namespaces/flyte/ingresses/flyte-core
      uid: 9cc9d62a-6e3d-4808-b98a-815f62408bf8
    spec:
      rules:
      - host: birdejo.live
        http:
          paths:
          - backend:
              serviceName: flyteadmin
              servicePort: 87
            path: /openapi
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteconsole
              servicePort: 80
            path: /console
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteconsole
              servicePort: 80
            path: /console/*
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /api
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /api/*
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /healthcheck
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /v1/*
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 87
            path: /openapi/*
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /.well-known
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /.well-known/*
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /login
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /login/*
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /logout
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /logout/*
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /callback
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /callback/*
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /me
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /config
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /config/*
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /oauth2
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 80
            path: /oauth2/*
            pathType: ImplementationSpecific
    status:
      loadBalancer:
        ingress:
        - hostname: <http://ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com|ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com>
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      annotations:
        <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>: '{"Type": "redirect", "RedirectConfig":
          { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
        <http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: HTTP2
        <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: <CERTIFICATE_ARN>
        <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
        <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTP": 80}, {"HTTPS":443}]'
        <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
        <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
        <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
        <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: service-flyte-helmchart
        <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
        <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
        <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
      creationTimestamp: "2021-12-27T01:48:25Z"
      generation: 1
      labels:
        <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
      name: flyte-core-grpc
      namespace: flyte
      resourceVersion: "131146"
      selfLink: /apis/extensions/v1beta1/namespaces/flyte/ingresses/flyte-core-grpc
      uid: 5512ef0b-eb59-41bd-a1f8-f45b9dfad290
    spec:
      rules:
      - host: birdejo.live
        http:
          paths:
          - backend:
              serviceName: flyteadmin
              servicePort: 81
            path: /flyteidl.service.AdminService
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 81
            path: /flyteidl.service.AdminService/*
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 81
            path: /flyteidl.service.AuthMetadataService
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 81
            path: /flyteidl.service.AuthMetadataService/*
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 81
            path: /flyteidl.service.IdentityService
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 81
            path: /flyteidl.service.IdentityService/*
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 81
            path: /grpc.health.v1.Health
            pathType: ImplementationSpecific
          - backend:
              serviceName: flyteadmin
              servicePort: 81
            path: /grpc.health.v1.Health/*
            pathType: ImplementationSpecific
    status:
      loadBalancer:
        ingress:
        - hostname: <http://ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com|ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com>
    y

    Yuvraj

    9 months ago
    Can you set the value of
    delegate: true
    and then re run the opta ? https://github.com/flyteorg/flyte/blob/master/opta/aws/env.yaml#L11
    e

    Eugene Cha

    9 months ago
    delegated is set to true and opta apply -c env.yaml doenst have any errors
    y

    Yuvraj

    9 months ago
    your ingress spec doesn't have correct certificate arn
    <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: <CERTIFICATE_ARN>
    p

    Prafulla Mahindrakar

    9 months ago
    @Yuvraj we dont have separate steps for updating these values in our docs for opta https://docs.flyte.org/en/latest/deployment/aws/opta.html Are there some additional steps needed to update this and can we point to the docs
    y

    Yuvraj

    9 months ago
    no i don't think so, opta should update it.
    e

    Eugene Cha

    9 months ago
    Ahhh
    p

    Prafulla Mahindrakar

    9 months ago
    Ok. @Yuvraj has this worked in the recent builds for our internal site. Might be a bug in opta if its not updating these values ?
    Also @Eugene Cha, can you try using flytectl command line and see if that works for you using the flyteadmin external-IP. You can get that from the following
    kubectl get service -n flyte
    e

    Eugene Cha

    9 months ago
    Yes
    p

    Prafulla Mahindrakar

    9 months ago
    Update ~/.flyte/config.yaml with the alb address and port 81 eg :
    admin:
      # For GRPC endpoints you might want to use dns:///flyte.myexample.com
      endpoint: <uuid>.<http://us-east-2.elb.amazonaws.com:81|us-east-2.elb.amazonaws.com:81>
      authType: Pkce
      insecure: true
    e

    Eugene Cha

    9 months ago
    is that supposed to be on my local or in the kubernetes cluster?
    p

    Prafulla Mahindrakar

    9 months ago
    This config would be on the local machine when using flytectl
    e

    Eugene Cha

    9 months ago
    got it
    Sorry for the late reply, but I have flytectl installed but i dont have a .flyte folder in ~
    y

    Yuvraj

    9 months ago
    please try this.
    flytectl config init --host <uuid>.<http://us-east-2.elb.amazonaws.com:81|us-east-2.elb.amazonaws.com:81> --insecure
    p

    Prafulla Mahindrakar

    8 months ago
    Also @Eugene Cha, we might want to check also your route53 hostedzones from the deleted cluster and if there are any left over records from the previous one. If yes , then please go ahead and manually delete those and do a rerun of opta deployment by having delegated as false in the first step and then turning it on and applying it again. If this doesn't work then we will add the opta team to take a deeper look at your issue. Also let us know if you were able to get to use the flyteadmin alb url to work with flytectl which will help us narrow down the problem to upper layers .
    e

    Eugene Cha

    8 months ago
    This is what I get when I kubectl get service -n flyte
    NAME                TYPE           CLUSTER-IP       EXTERNAL-IP                                                                    PORT(S)                                                  AGE
    datacatalog         LoadBalancer   172.20.54.245    <http://ac48efd29e98942b4aa4a430ead8cb9e-923941080.ap-northeast-2.elb.amazonaws.com|ac48efd29e98942b4aa4a430ead8cb9e-923941080.ap-northeast-2.elb.amazonaws.com>    8089:32568/TCP,88:31929/TCP,89:31078/TCP                 36h
    flyte-pod-webhook   ClusterIP      172.20.236.53    <none>                                                                         443/TCP                                                  36h
    flyteadmin          LoadBalancer   172.20.163.192   <http://ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com|ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com>     80:32626/TCP,81:30177/TCP,87:31898/TCP,10254:32737/TCP   36h
    flyteconsole        LoadBalancer   172.20.157.1     <http://a31945de8f10c440a85b8b3491ced10a-1691719649.ap-northeast-2.elb.amazonaws.com|a31945de8f10c440a85b8b3491ced10a-1691719649.ap-northeast-2.elb.amazonaws.com>   80:32150/TCP                                             36h
    This is what I get when I put the flyteadmin url in config
    โฏ flytectl config init --host <http://ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81|ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81> --insecure
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
    {"json":{},"level":"error","msg":"failed to initialize token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: missing address\"","ts":"2021-12-28T23:37:58+09:00"}
    {"json":{},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2021-12-28T23:37:58+09:00"}
    {"json":{},"level":"info","msg":"Initialized Admin client","ts":"2021-12-28T23:37:58+09:00"}
    Error: Please use a valid endpoint
    {"json":{},"level":"error","msg":"Please use a valid endpoint","ts":"2021-12-28T23:37:58+09:00"}
    Is there supposed to be one hosted zone in route 53? There's three. One's with the correct NS. One has the previous NS. And one I think is from the destroyed cluster but I'm not sure
    the ingress hostname for the load balancer is currently ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com. is it supposed to be ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com which is the flyteadmin url?
    y

    Yuvraj

    8 months ago
    I just tested your endpoint and it worked for me,
    20:30:48 โžœ cookbook flytectl config init --host='<http://ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81|ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81>' --insecure
    This action will overwrite an existing config file at [/Users/yuvraj/.flyte/config.yaml]. Do you want to continue? [y/n]: y
    Init flytectl config file at [/Users/yuvraj/.flyte/config.yaml]%                                                                                                                                                                           20:31:46 โžœ cookbook flytectl get projects                                                                                                                      
     --------------- --------------- --------------------------- 
    | ID            | NAME          | DESCRIPTION               |
     --------------- --------------- --------------------------- 
    | flyteexamples | flyteexamples | flyteexamples description |
     --------------- --------------- --------------------------- 
    | flytetester   | flytetester   | flytetester description   |
     --------------- --------------- --------------------------- 
    | flytesnacks   | flytesnacks   | flytesnacks description   |
     --------------- --------------- --------------------------- 
    3 rows
    In config command we added the validation recently and it's a bug, I will create a fix for that, For now you can use this config for validation
    ~/.flyte/config.yaml
    admin:
      # For GRPC endpoints you might want to use dns:///flyte.myexample.com
      endpoint: dns:///ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81
      authType: Pkce
      insecure: true
    logger:
      show-source: true
      level: 0
    e

    Eugene Cha

    8 months ago
    Hmmm. I copy pasted flytectl config init --host='ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81' --insecure from your code and i get the same please use a valid endpoint
    For config.yaml do I just create a new config.yaml file in ~/.flyte?
    Sorry for the dumb questions I'm kinda lost
    y

    Yuvraj

    8 months ago
    Please manually add the config at
    ~/.flyte/config.yaml
    e

    Eugene Cha

    8 months ago
    Ah so I don't have a config.yaml in my ~/.flyte folder
    y

    Yuvraj

    8 months ago
    create one
    I will create a pr for bug fix
    e

    Eugene Cha

    8 months ago
    ok
    * cannot parse 'level' as int: strconv.ParseInt: parsing "0%": invalid syntax src="main.go:13"
    deleted the % and I still get a valid endpoint error
    y

    Yuvraj

    8 months ago
    my bad in config level is 0, Please check
    e

    Eugene Cha

    8 months ago
    Would it be easier to opta destroy, go through the aws resources make sure everything is gone and then opta apply again?
    y

    Yuvraj

    8 months ago
    Replace config and then run this command
    flytectl get projects
    e

    Eugene Cha

    8 months ago
    y

    Yuvraj

    8 months ago
    Now you are connected to flyte cluster using flytectl, For console we need to fix ingress
    e

    Eugene Cha

    8 months ago
    awesome
    for ingress can I create a yaml file where I just replace the <CERTIFICATE_ARN> with the actual ARN in the output of kubectl get ingress -n flyte flyte-core -o yaml then kubectl apply -f name-of-yaml-file.yaml
    y

    Yuvraj

    8 months ago
    you don't need certificate_arn because opta use nginx for ingress
    e

    Eugene Cha

    8 months ago
    hmm. what should i do to fix the ingress?
    y

    Yuvraj

    8 months ago
    Ketan (kumare3)

    Ketan (kumare3)

    8 months ago
    Cc @Eugene Cha can we make this a GitHub discussion and copy the solution there - for others to understand ๐Ÿ™
    e

    Eugene Cha

    8 months ago
    that's so weird. It's not working for me
    y

    Yuvraj

    8 months ago
    e

    Eugene Cha

    8 months ago
    what the...? I tried on a different computer as well and it's not working there either
    i'm going to go to bed, but thank Yuvraj for the help. I'll try and see if other people can access tomorrow
    p

    Prafulla Mahindrakar

    8 months ago
    @Eugene Cha I am able to access the link aswell .it might be some local dns cache issue on your end .
    Ketan (kumare3)

    Ketan (kumare3)

    8 months ago
    Btw me too, in seattle
    e

    Eugene Cha

    8 months ago
    thatโ€™s so weird. i can access from my phone on my phone network but not from my computers on my wifi network
    p

    Prafulla Mahindrakar

    8 months ago
    Can you flush your local dns cache
    e

    Eugene Cha

    8 months ago
    i tried flushing my local dns cache and it still doesn't work. I can access via my phone. Can't access on my ipad using the wifi or the phone network
    p

    Prafulla Mahindrakar

    8 months ago
    That seems strange even i can't access now. BTW There should be only one hosted zone for flyte . Do you mind redoing opta again by cleaning up the setup using opta destroy and then also checking if the hosted zones are cleared up and if not manually clear them for now and we will file a bug with opta team .
    e

    Eugene Cha

    8 months ago
    yes
    i'll opta destroy and redo