https://flyte.org logo
Join Slack
Powered by
Hi all. I originally had flyte working, but had to...
# ask-the-community
e
Hi all. I originally had flyte working, but had to delete the cluster and reinstall. I passed the deployment stage and set it using this console (https://birdejo.live). Unfortunately the console doesnt seem to work. Any ideas on how to diagnose what the issue is?
k
What do you mean not working?
Can you check the pods in the k8s cluster- Flyte namespace
cc @Prafulla Mahindrakar / @Yuvraj incase @Eugene Cha happens to ask again
✅ 1
e
On the admin pods I get a no update handler registered as well as on the webhook pods
Copy code
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [flyteadmin] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [scheduler] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [remotedata] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [cluster_resources] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [qualityofservice] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [registration] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [notifications] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [domains] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [task_resources] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [secrets] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [queues] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [namespace_mapping] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [task_type_whitelist] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [server] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.catalogcache] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.k8s] updated. No update handler registered.","ts":"2021-12-27T01:48:57Z"}
k
That’s ok, but the pods are up and not crashing right?
p
That shouldn't cause the issue you are seeing . Can you paste details if all the flyte pods are runnings and also
Also can you check the ingress details aswell using kubectl get ingress -n flyte
e
pods are up and not crashing
message has been deleted
ingress seems to be working
basically can't connect to the console
p
Can you also dump the ingress details. kubectl get ingress -n flyte flyte-core -o yaml kubectl get ingress -n flyte flyte-core-grpc -o yaml
e
Copy code
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: <CERTIFICATE_ARN>
    <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
    <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTP": 80}, {"HTTPS":443}]'
    <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
    <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: service-flyte-helmchart
    <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
  creationTimestamp: "2021-12-27T01:48:25Z"
  generation: 1
  labels:
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
  name: flyte-core
  namespace: flyte
  resourceVersion: "131147"
  selfLink: /apis/extensions/v1beta1/namespaces/flyte/ingresses/flyte-core
  uid: 9cc9d62a-6e3d-4808-b98a-815f62408bf8
spec:
  rules:
  - host: birdejo.live
    http:
      paths:
      - backend:
          serviceName: flyteadmin
          servicePort: 87
        path: /openapi
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteconsole
          servicePort: 80
        path: /console
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteconsole
          servicePort: 80
        path: /console/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /api
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /api/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /healthcheck
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /v1/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 87
        path: /openapi/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /.well-known
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /.well-known/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /login
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /login/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /logout
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /logout/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /callback
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /callback/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /me
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /config
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /config/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /oauth2
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 80
        path: /oauth2/*
        pathType: ImplementationSpecific
status:
  loadBalancer:
    ingress:
    - hostname: <http://ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com|ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com>
Copy code
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    <http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: HTTP2
    <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: <CERTIFICATE_ARN>
    <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
    <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTP": 80}, {"HTTPS":443}]'
    <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
    <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: service-flyte-helmchart
    <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
  creationTimestamp: "2021-12-27T01:48:25Z"
  generation: 1
  labels:
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
  name: flyte-core-grpc
  namespace: flyte
  resourceVersion: "131146"
  selfLink: /apis/extensions/v1beta1/namespaces/flyte/ingresses/flyte-core-grpc
  uid: 5512ef0b-eb59-41bd-a1f8-f45b9dfad290
spec:
  rules:
  - host: birdejo.live
    http:
      paths:
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /flyteidl.service.AdminService
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /flyteidl.service.AdminService/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /flyteidl.service.AuthMetadataService
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /flyteidl.service.AuthMetadataService/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /flyteidl.service.IdentityService
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /flyteidl.service.IdentityService/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /grpc.health.v1.Health
        pathType: ImplementationSpecific
      - backend:
          serviceName: flyteadmin
          servicePort: 81
        path: /grpc.health.v1.Health/*
        pathType: ImplementationSpecific
status:
  loadBalancer:
    ingress:
    - hostname: <http://ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com|ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com>
y
Can you set the value of 
delegate: true
and then re run the opta ? https://github.com/flyteorg/flyte/blob/master/opta/aws/env.yaml#L11
e
delegated is set to true and opta apply -c env.yaml doenst have any errors
y
your ingress spec doesn't have correct certificate arn 
<http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: <CERTIFICATE_ARN>
p
@Yuvraj we dont have separate steps for updating these values in our docs for opta https://docs.flyte.org/en/latest/deployment/aws/opta.html Are there some additional steps needed to update this and can we point to the docs
y
no i don't think so, opta should update it.
e
Ahhh
p
Ok. @Yuvraj has this worked in the recent builds for our internal site. Might be a bug in opta if its not updating these values ?
👍 1
Also @Eugene Cha, can you try using flytectl command line and see if that works for you using the flyteadmin external-IP. You can get that from the following
Copy code
kubectl get service -n flyte
e
Yes
p
Update ~/.flyte/config.yaml with the alb address and port 81 eg :
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: <uuid>.<http://us-east-2.elb.amazonaws.com:81|us-east-2.elb.amazonaws.com:81>
  authType: Pkce
  insecure: true
e
is that supposed to be on my local or in the kubernetes cluster?
p
This config would be on the local machine when using flytectl
e
got it
Sorry for the late reply, but I have flytectl installed but i dont have a .flyte folder in ~
y
please try this. 
flytectl config init --host <uuid>.<http://us-east-2.elb.amazonaws.com:81|us-east-2.elb.amazonaws.com:81> --insecure
p
Also @Eugene Cha, we might want to check also your route53 hostedzones from the deleted cluster and if there are any left over records from the previous one. If yes , then please go ahead and manually delete those and do a rerun of opta deployment by having delegated as false in the first step and then turning it on and applying it again. If this doesn't work then we will add the opta team to take a deeper look at your issue. Also let us know if you were able to get to use the flyteadmin alb url to work with flytectl which will help us narrow down the problem to upper layers .
e
This is what I get when I kubectl get service -n flyte
Copy code
NAME                TYPE           CLUSTER-IP       EXTERNAL-IP                                                                    PORT(S)                                                  AGE
datacatalog         LoadBalancer   172.20.54.245    <http://ac48efd29e98942b4aa4a430ead8cb9e-923941080.ap-northeast-2.elb.amazonaws.com|ac48efd29e98942b4aa4a430ead8cb9e-923941080.ap-northeast-2.elb.amazonaws.com>    8089:32568/TCP,88:31929/TCP,89:31078/TCP                 36h
flyte-pod-webhook   ClusterIP      172.20.236.53    <none>                                                                         443/TCP                                                  36h
flyteadmin          LoadBalancer   172.20.163.192   <http://ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com|ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com>     80:32626/TCP,81:30177/TCP,87:31898/TCP,10254:32737/TCP   36h
flyteconsole        LoadBalancer   172.20.157.1     <http://a31945de8f10c440a85b8b3491ced10a-1691719649.ap-northeast-2.elb.amazonaws.com|a31945de8f10c440a85b8b3491ced10a-1691719649.ap-northeast-2.elb.amazonaws.com>   80:32150/TCP                                             36h
This is what I get when I put the flyteadmin url in config
Copy code
❯ flytectl config init --host <http://ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81|ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81> --insecure
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
{"json":{},"level":"error","msg":"failed to initialize token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: missing address\"","ts":"2021-12-28T23:37:58+09:00"}
{"json":{},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2021-12-28T23:37:58+09:00"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2021-12-28T23:37:58+09:00"}
Error: Please use a valid endpoint
{"json":{},"level":"error","msg":"Please use a valid endpoint","ts":"2021-12-28T23:37:58+09:00"}
Is there supposed to be one hosted zone in route 53? There's three. One's with the correct NS. One has the previous NS. And one I think is from the destroyed cluster but I'm not sure
the ingress hostname for the load balancer is currently ab4327749f98949d894361022a7d41eb-7d04531a84863ee6.elb.ap-northeast-2.amazonaws.com. is it supposed to be ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com which is the flyteadmin url?
y
I just tested your endpoint and it worked for me,
Copy code
20:30:48 ➜ cookbook flytectl config init --host='<http://ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81|ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81>' --insecure
This action will overwrite an existing config file at [/Users/yuvraj/.flyte/config.yaml]. Do you want to continue? [y/n]: y
Init flytectl config file at [/Users/yuvraj/.flyte/config.yaml]%                                                                                                                                                                           20:31:46 ➜ cookbook flytectl get projects                                                                                                                      
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester   | flytetester   | flytetester description   |
 --------------- --------------- --------------------------- 
| flytesnacks   | flytesnacks   | flytesnacks description   |
 --------------- --------------- --------------------------- 
3 rows
In config command we added the validation recently and it's a bug, I will create a fix for that, For now you can use this config for validation 
~/.flyte/config.yaml
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
e
Hmmm. I copy pasted flytectl config init --host='ae7fc964d0d1b498fb0d955e9d0f5977-80905779.ap-northeast-2.elb.amazonaws.com:81' --insecure from your code and i get the same please use a valid endpoint
For config.yaml do I just create a new config.yaml file in ~/.flyte?
Sorry for the dumb questions I'm kinda lost
y
Please manually add the config at 
~/.flyte/config.yaml
e
Ah so I don't have a config.yaml in my ~/.flyte folder
y
create one
I will create a pr for bug fix
e
ok
* cannot parse 'level' as int: strconv.ParseInt: parsing "0%": invalid syntax src="main.go:13"
deleted the % and I still get a valid endpoint error
y
my bad in config level is 0, Please check
e
Would it be easier to opta destroy, go through the aws resources make sure everything is gone and then opta apply again?
y
Replace config and then run this command
Copy code
flytectl get projects
e
message has been deleted
🙌 1
y
Now you are connected to flyte cluster using flytectl, For console we need to fix ingress
e
awesome
for ingress can I create a yaml file where I just replace the <CERTIFICATE_ARN> with the actual ARN in the output of kubectl get ingress -n flyte flyte-core -o yaml then kubectl apply -f name-of-yaml-file.yaml
y
you don't need certificate_arn because opta use nginx for ingress
e
hmm. what should i do to fix the ingress?
y
k
Cc @Eugene Cha can we make this a GitHub discussion and copy the solution there - for others to understand 🙏
👍 1
e
that's so weird. It's not working for me
y
message has been deleted
e
what the...? I tried on a different computer as well and it's not working there either
message has been deleted
i'm going to go to bed, but thank Yuvraj for the help. I'll try and see if other people can access tomorrow
👍 1
p
@Eugene Cha I am able to access the link aswell .it might be some local dns cache issue on your end .
k
Btw me too, in seattle
e
that’s so weird. i can access from my phone on my phone network but not from my computers on my wifi network
p
Can you flush your local dns cache
e
i tried flushing my local dns cache and it still doesn't work. I can access via my phone. Can't access on my ipad using the wifi or the phone network
p
That seems strange even i can't access now. BTW There should be only one hosted zone for flyte . Do you mind redoing opta again by cleaning up the setup using opta destroy and then also checking if the hosted zones are cleared up and if not manually clear them for now and we will file a bug with opta team .
e
yes
i'll opta destroy and redo
198 Views
Open in Slack
PreviousNext
Powered by