https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
e

Eugene Cha

12/27/2021, 2:43 AM
Hi all. I originally had flyte working, but had to delete the cluster and reinstall. I passed the deployment stage and set it using this console (https://birdejo.live). Unfortunately the console doesnt seem to work. Any ideas on how to diagnose what the issue is?
k

Ketan (kumare3)

12/27/2021, 2:59 AM
What do you mean not working?
Can you check the pods in the k8s cluster- Flyte namespace
cc @Prafulla Mahindrakar / @Yuvraj incase @Eugene Cha happens to ask again
✅ 1
e

Eugene Cha

12/27/2021, 6:48 AM
On the admin pods I get a no update handler registered as well as on the webhook pods
Copy code
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [flyteadmin] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [scheduler] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [remotedata] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [cluster_resources] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [qualityofservice] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [registration] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [notifications] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [domains] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [task_resources] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [secrets] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [queues] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [namespace_mapping] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [task_type_whitelist] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [server] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.catalogcache] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.k8s] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
k

Ketan (kumare3)

12/27/2021, 6:50 AM
That’s ok, but the pods are up and not crashing right?
p

Prafulla Mahindrakar

12/27/2021, 6:50 AM
That shouldn't cause the issue you are seeing . Can you paste details if all the flyte pods are runnings and also
Also can you check the ingress details aswell using kubectl get ingress -n flyte
e

Eugene Cha

12/27/2021, 7:18 AM
pods are up and not crashing
message has been deleted
ingress seems to be working
basically can't connect to the console
p

Prafulla Mahindrakar

12/27/2021, 7:23 AM
Can you also dump the ingress details. kubectl get ingress -n flyte flyte-core -o yaml kubectl get ingress -n flyte flyte-core-grpc -o yaml
e

Eugene Cha

12/27/2021, 7:23 AM
Copy code
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: <CERTIFICATE_ARN>
    <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
    <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTP": 80}, {"HTTPS":443}]'
    <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
    <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: service-flyte-helmchart
    <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
  creationTimestamp: "2021-12-27T01:48:25Z"
  generation: 1
  labels:
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
  name: flyte-core
  namespace: flyte
  resourceVersion: "131147"
  selfLink: /apis/extensions/v1beta1/namespaces/flyte/ingresses/flyte-core
  uid: 9cc9d62a-6e3d-4808-b98a-815f62408bf8
spec:
  rules:
  - host: birdejo.live
    http:
      paths:
      - backend:
          serviceName: flyteadmin
          servicePort: 87
        path: /openapi
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteconsole
          servicePort: 80
        path: /console
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteconsole
          servicePort: 80
        path: /console/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /api
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /api/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /healthcheck
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /v1/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 87
        path: /openapi/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /.well-known
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /.well-known/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /login
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /login/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /logout
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /logout/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /callback
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /callback/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /me
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /config
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /config/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /oauth2
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /oauth2/*
        pathType: ImplementationSpecific
status:
  loadBalancer:
    ingress:
    - hostname: <http://ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com|ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com>
Copy code
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    <http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: HTTP2
    <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: <CERTIFICATE_ARN>
    <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
    <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTP": 80}, {"HTTPS":443}]'
    <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
    <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: service-flyte-helmchart
    <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
  creationTimestamp: "2021-12-27T01:48:25Z"
  generation: 1
  labels:
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
  name: flyte-core-grpc
  namespace: flyte
  resourceVersion: "131146"
  selfLink: /apis/extensions/v1beta1/namespaces/flyte/ingresses/flyte-core-grpc
  uid: 5512ef0b-eb59-41bd-a1f8-f45b9dfad290
spec:
  rules:
  - host: birdejo.live
    http:
      paths:
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /flyteidl.service.AdminService
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /flyteidl.service.AdminService/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /flyteidl.service.AuthMetadataService
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /flyteidl.service.AuthMetadataService/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /flyteidl.service.IdentityService
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /flyteidl.service.IdentityService/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /grpc.health.v1.Health
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /grpc.health.v1.Health/*
        pathType: ImplementationSpecific
status:
  loadBalancer:
    ingress:
    - hostname: <http://ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com|ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com>
y

Yuvraj

12/27/2021, 7:25 AM
Can you set the value of 
delegate: true
and then re run the opta ? https://github.com/flyteorg/flyte/blob/master/opta/aws/env.yaml#L11
e

Eugene Cha

12/27/2021, 7:26 AM
delegated is set to true and opta apply -c env.yaml doenst have any errors
y

Yuvraj

12/27/2021, 7:30 AM
your ingress spec doesn't have correct certificate arn 
<http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: <CERTIFICATE_ARN>
p

Prafulla Mahindrakar

12/27/2021, 7:32 AM
@Yuvraj we dont have separate steps for updating these values in our docs for opta https://docs.flyte.org/en/latest/deployment/aws/opta.html Are there some additional steps needed to update this and can we point to the docs
y

Yuvraj

12/27/2021, 7:34 AM
no i don't think so, opta should update it.
e

Eugene Cha

12/27/2021, 7:36 AM
Ahhh
p

Prafulla Mahindrakar

12/27/2021, 7:37 AM
Ok. @Yuvraj has this worked in the recent builds for our internal site. Might be a bug in opta if its not updating these values ?
👍 1
Also @Eugene Cha, can you try using flytectl command line and see if that works for you using the flyteadmin external-IP. You can get that from the following
Copy code
kubectl get service -n flyte
e

Eugene Cha

12/27/2021, 7:43 AM
Yes
p

Prafulla Mahindrakar

12/27/2021, 7:43 AM
Update ~/.flyte/config.yaml with the alb address and port 81 eg :
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: <uuid>.<http://us-east-2.elb.amazonaws.com:81|us-east-2.elb.amazonaws.com:81>
  authType: Pkce
  insecure: true
e

Eugene Cha

12/27/2021, 9:35 AM
is that supposed to be on my local or in the kubernetes cluster?
p

Prafulla Mahindrakar

12/27/2021, 9:36 AM
This config would be on the local machine when using flytectl
e

Eugene Cha

12/27/2021, 9:37 AM
got it
Sorry for the late reply, but I have flytectl installed but i dont have a .flyte folder in ~
y

Yuvraj

12/27/2021, 3:16 PM
please try this. 
flytectl config init --host <uuid>.<http://us-east-2.elb.amazonaws.com:81|us-east-2.elb.amazonaws.com:81> --insecure
p

Prafulla Mahindrakar

12/28/2021, 6:25 AM
Also @Eugene Cha, we might want to check also your route53 hostedzones from the deleted cluster and if there are any left over records from the previous one. If yes , then please go ahead and manually delete those and do a rerun of opta deployment by having delegated as false in the first step and then turning it on and applying it again. If this doesn't work then we will add the opta team to take a deeper look at your issue. Also let us know if you were able to get to use the flyteadmin alb url to work with flytectl which will help us narrow down the problem to upper layers .
e

Eugene Cha

12/28/2021, 2:42 PM
This is what I get when I kubectl get service -n flyte
Copy code
NAME                TYPE           CLUSTER-IP       EXTERNAL-IP                                                                    PORT(S)                                                  AGE
datacatalog         LoadBalancer   172.20.54.245    <http://ac48efd29e98942b4aa4a430ead8cb9e-923941080.ap-northeast-2.elb.amazonaws.com|ac48efd29e98942b4aa4a430ead8cb9e-923941080.ap-northeast-2.elb.amazonaws.com>    8089:32568/TCP,88:31929/TCP,89:31078/TCP                 36h
flyte-pod-webhook   ClusterIP      172.20.236.53    <none>                                                                         443/TCP                                                  36h
flyteadmin          LoadBalancer   172.20.163.192   <http://ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com|ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com>     80:32626/TCP,81:30177/TCP,87:31898/TCP,10254:32737/TCP   36h
flyteconsole        LoadBalancer   172.20.157.1     <http://a31945de8f10c440a85b8b3491ced10a-1691719649.ap-northeast-2.elb.amazonaws.com|a31945de8f10c440a85b8b3491ced10a-1691719649.ap-northeast-2.elb.amazonaws.com>   80:32150/TCP                                             36h
This is what I get when I put the flyteadmin url in config
Copy code
❯ flytectl config init --host <http://ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81|ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81> --insecure
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
{"json":{},"level":"error","msg":"failed to initialize token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: missing address\"","ts":"2021-12-28T23:37:58+09:00"}
{"json":{},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2021-12-28T23:37:58+09:00"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2021-12-28T23:37:58+09:00"}
Error: Please use a valid endpoint
{"json":{},"level":"error","msg":"Please use a valid endpoint","ts":"2021-12-28T23:37:58+09:00"}
Is there supposed to be one hosted zone in route 53? There's three. One's with the correct NS. One has the previous NS. And one I think is from the destroyed cluster but I'm not sure
the ingress hostname for the load balancer is currently ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com. is it supposed to be ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com which is the flyteadmin url?
y

Yuvraj

12/28/2021, 3:05 PM
I just tested your endpoint and it worked for me,
Copy code
20:30:48 ➜ cookbook flytectl config init --host='<http://ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81|ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81>' --insecure
This action will overwrite an existing config file at [/Users/yuvraj/.flyte/config.yaml]. Do you want to continue? [y/n]: y
Init flytectl config file at [/Users/yuvraj/.flyte/config.yaml]%                                                                                                                                                                           20:31:46 ➜ cookbook flytectl get projects                                                                                                                      
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester   | flytetester   | flytetester description   |
 --------------- --------------- --------------------------- 
| flytesnacks   | flytesnacks   | flytesnacks description   |
 --------------- --------------- --------------------------- 
3 rows
In config command we added the validation recently and it's a bug, I will create a fix for that, For now you can use this config for validation 
~/.flyte/config.yaml
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
e

Eugene Cha

12/28/2021, 3:14 PM
Hmmm. I copy pasted flytectl config init --host='ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81' --insecure from your code and i get the same please use a valid endpoint
For config.yaml do I just create a new config.yaml file in ~/.flyte?
Sorry for the dumb questions I'm kinda lost
y

Yuvraj

12/28/2021, 3:15 PM
Please manually add the config at 
~/.flyte/config.yaml
e

Eugene Cha

12/28/2021, 3:15 PM
Ah so I don't have a config.yaml in my ~/.flyte folder
y

Yuvraj

12/28/2021, 3:15 PM
create one
I will create a pr for bug fix
e

Eugene Cha

12/28/2021, 3:16 PM
ok
* cannot parse 'level' as int: strconv.ParseInt: parsing "0%": invalid syntax src="main.go:13"
deleted the % and I still get a valid endpoint error
y

Yuvraj

12/28/2021, 3:18 PM
my bad in config level is 0, Please check
e

Eugene Cha

12/28/2021, 3:18 PM
Would it be easier to opta destroy, go through the aws resources make sure everything is gone and then opta apply again?
y

Yuvraj

12/28/2021, 3:18 PM
Replace config and then run this command
Copy code
flytectl get projects
e

Eugene Cha

12/28/2021, 3:19 PM
message has been deleted
🙌 1
y

Yuvraj

12/28/2021, 3:19 PM
Now you are connected to flyte cluster using flytectl, For console we need to fix ingress
e

Eugene Cha

12/28/2021, 3:19 PM
awesome
for ingress can I create a yaml file where I just replace the <CERTIFICATE_ARN> with the actual ARN in the output of kubectl get ingress -n flyte flyte-core -o yaml then kubectl apply -f name-of-yaml-file.yaml
y

Yuvraj

12/28/2021, 3:30 PM
you don't need certificate_arn because opta use nginx for ingress
e

Eugene Cha

12/28/2021, 3:30 PM
hmm. what should i do to fix the ingress?
y

Yuvraj

12/28/2021, 3:33 PM
k

Ketan (kumare3)

12/28/2021, 3:34 PM
Cc @Eugene Cha can we make this a GitHub discussion and copy the solution there - for others to understand 🙏
👍 1
e

Eugene Cha

12/28/2021, 3:34 PM
that's so weird. It's not working for me
y

Yuvraj

12/28/2021, 3:35 PM
message has been deleted
e

Eugene Cha

12/28/2021, 3:35 PM
what the...? I tried on a different computer as well and it's not working there either
message has been deleted
i'm going to go to bed, but thank Yuvraj for the help. I'll try and see if other people can access tomorrow
👍 1
p

Prafulla Mahindrakar

12/29/2021, 4:53 AM
@Eugene Cha I am able to access the link aswell .it might be some local dns cache issue on your end .
k

Ketan (kumare3)

12/29/2021, 4:55 AM
Btw me too, in seattle
e

Eugene Cha

12/29/2021, 7:25 AM
that’s so weird. i can access from my phone on my phone network but not from my computers on my wifi network
p

Prafulla Mahindrakar

12/29/2021, 7:26 AM
Can you flush your local dns cache
e

Eugene Cha

12/30/2021, 8:36 AM
i tried flushing my local dns cache and it still doesn't work. I can access via my phone. Can't access on my ipad using the wifi or the phone network
p

Prafulla Mahindrakar

12/30/2021, 8:44 AM
That seems strange even i can't access now. BTW There should be only one hosted zone for flyte . Do you mind redoing opta again by cleaning up the setup using opta destroy and then also checking if the hosted zones are cleared up and if not manually clear them for now and we will file a bug with opta team .
e

Eugene Cha

12/30/2021, 8:57 AM
yes
i'll opta destroy and redo
36 Views
Powered by