Hey team we deployed flyte to eks cluster now when we are ru Flyte #flyte-support

Hey team we deployed flyte to eks cluster now when...

brash-piano-42461

09/26/2023, 8:10 AM

Hey team we deployed flyte to eks cluster now when we are running it shows status unknown can anyone help me with it. Thanks!

brash-piano-42461

09/26/2023, 9:35 AM

average-finland-92144

09/26/2023, 10:41 AM

HI @brash-piano-42461 can you share: 1. Status of Flyte's pods

kubectl get po -n flyte

2. Status of tasks' Pods (assuming you're using the example workflow with no changes)

kubectl get po -n flytesnackes-development

If there's some component Pending or failed, could you run a

kubectl describe po <pod-name>

brash-piano-42461

09/26/2023, 10:42 AM

all are running

brash-piano-42461

09/26/2023, 10:42 AM

@average-finland-92144

brash-piano-42461

09/26/2023, 10:43 AM

Copy code

kubectl get po -n flytesnacks-development
No resources found in flytesnacks-development namespace.

average-finland-92144

09/26/2023, 10:43 AM

ok, are you running

flyte-binary

brash-piano-42461

09/26/2023, 10:44 AM

yes

brash-piano-42461

09/26/2023, 10:44 AM

Copy code

ubectl get po -n flyte
NAME                            READY   STATUS    RESTARTS       AGE
flyte-binary-65fff8bf4b-n86kt   1/1     Running   9 (156m ago)   3h22m

average-finland-92144

09/26/2023, 10:44 AM

please get logs from the Flyte pod

kubectl logs -n flyte flyte-binary-65fff8bf4b-n86kt

brash-piano-42461

09/26/2023, 10:44 AM

i followed this doc https://github.com/unionai-oss/deploy-flyte/blob/main/environments/aws/README.md

✅ 1

brash-piano-42461

09/26/2023, 10:45 AM

Copy code

2023/09/26 10:44:25 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/callbacks.go:134 record not found
[0.566ms] [rows:0] SELECT * FROM "resources" WHERE resource_type = 'CLUSTER_RESOURCE' AND domain IN ('','staging') AND project IN ('','flytesnacks') AND workflow IN ('') AND launch_plan IN ('') ORDER BY priority desc,"resources"."id" LIMIT 1
{"json":{"src":"controller.go:297"},"level":"debug","msg":"syncing namespace [flytesnacks-staging]: ignoring unrecognized filetype [..2023_09_26_07_20_24.1701649631]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:297"},"level":"debug","msg":"syncing namespace [flytesnacks-staging]: ignoring unrecognized filetype [..data]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:477"},"level":"debug","msg":"successfully read template config file [001_namespace.yaml]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:329"},"level":"debug","msg":"Attempting to create resource [Namespace] in cluster [] for namespace [flytesnacks-staging]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:337"},"level":"debug","msg":"Type [Namespace] in namespace [flytesnacks-staging] already exists - attempting update instead","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:372"},"level":"info","msg":"Resource [Namespace] in namespace [flytesnacks-staging] is not modified","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:477"},"level":"debug","msg":"successfully read template config file [002_serviceaccount.yaml]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:329"},"level":"debug","msg":"Attempting to create resource [ServiceAccount] in cluster [] for namespace [flytesnacks-staging]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:337"},"level":"debug","msg":"Type [ServiceAccount] in namespace [flytesnacks-staging] already exists - attempting update instead","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:372"},"level":"info","msg":"Resource [ServiceAccount] in namespace [flytesnacks-staging] is not modified","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:607"},"level":"debug","msg":"Successfully created kubernetes resources for [flytesnacks-staging]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:611"},"level":"info","msg":"Completed cluster resource creation loop for namespace [flytesnacks-staging] with stats: [{Created:0 Updated:0 AlreadyThere:2 Errored:0}]","ts":"2023-09-26T10:44:25Z"}

2023/09/26 10:44:25 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/callbacks.go:134 record not found
[0.611ms] [rows:0] SELECT * FROM "resources" WHERE resource_type = 'CLUSTER_RESOURCE' AND domain IN ('','production') AND project IN ('','flytesnacks') AND workflow IN ('') AND launch_plan IN ('') ORDER BY priority desc,"resources"."id" LIMIT 1
{"json":{"src":"controller.go:297"},"level":"debug","msg":"syncing namespace [flytesnacks-production]: ignoring unrecognized filetype [..2023_09_26_07_20_24.1701649631]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:297"},"level":"debug","msg":"syncing namespace [flytesnacks-production]: ignoring unrecognized filetype [..data]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:477"},"level":"debug","msg":"successfully read template config file [001_namespace.yaml]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:329"},"level":"debug","msg":"Attempting to create resource [Namespace] in cluster [] for namespace [flytesnacks-production]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:337"},"level":"debug","msg":"Type [Namespace] in namespace [flytesnacks-production] already exists - attempting update instead","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:372"},"level":"info","msg":"Resource [Namespace] in namespace [flytesnacks-production] is not modified","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:477"},"level":"debug","msg":"successfully read template config file [002_serviceaccount.yaml]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:329"},"level":"debug","msg":"Attempting to create resource [ServiceAccount] in cluster [] for namespace [flytesnacks-production]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:337"},"level":"debug","msg":"Type [ServiceAccount] in namespace [flytesnacks-production] already exists - attempting update instead","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:372"},"level":"info","msg":"Resource [ServiceAccount] in namespace [flytesnacks-production] is not modified","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:607"},"level":"debug","msg":"Successfully created kubernetes resources for [flytesnacks-production]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:611"},"level":"info","msg":"Completed cluster resource creation loop for namespace [flytesnacks-production] with stats: [{Created:0 Updated:0 AlreadyThere:2 Errored:0}]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:615"},"level":"info","msg":"Completed cluster resource creation loop with stats: [{Created:0 Updated:0 AlreadyThere:6 Errored:0}]","ts":"2023-09-26T10:44:25Z"}
{"json":{"src":"controller.go:633"},"level":"info","msg":"Successfully completed cluster resource creation loop","ts":"2023-09-26T10:44:25Z"}

2023/09/26 10:44:26 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/callbacks.go:134
[0.586ms] [rows:0] SELECT * FROM "schedulable_entities"
{"json":{"src":"controller.go:160"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/f678b4cd6aab741aba5b]","ts":"2023-09-26T10:44:26Z"}
{"json":{"src":"controller.go:160"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fac0ae320b8614b889e6]","ts":"2023-09-26T10:44:26Z"}
{"json":{"src":"controller.go:160"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fe884109a2062487185a]","ts":"2023-09-26T10:44:26Z"}
{"json":{"src":"controller.go:160"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/arqddn7q6q99k6ltglwb]","ts":"2023-09-26T10:44:26Z"}
{"json":{"src":"controller.go:160"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fe884109a2062487185a]","ts":"2023-09-26T10:44:36Z"}
{"json":{"src":"controller.go:160"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/arqddn7q6q99k6ltglwb]","ts":"2023-09-26T10:44:36Z"}
{"json":{"src":"controller.go:160"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/f678b4cd6aab741aba5b]","ts":"2023-09-26T10:44:36Z"}
{"json":{"src":"controller.go:160"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fac0ae320b8614b889e6]","ts":"2023-09-26T10:44:36Z"}
{"json":{"src":"controller.go:160"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/f678b4cd6aab741aba5b]","ts":"2023-09-26T10:44:46Z"}
{"json":{"src":"controller.go:160"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fac0ae320b8614b889e6]","ts":"2023-09-26T10:44:46Z"}
{"json":{"src":"controller.go:160"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fe884109a2062487185a]","ts":"2023-09-26T10:44:46Z"}
{"json":{"src":"controller.go:160"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/arqddn7q6q99k6ltglwb]","ts":"2023-09-26T10:44:46Z"}

👀 1

brash-piano-42461

09/26/2023, 10:46 AM

do i need to create

flytesnacks-development

namespace?

brash-piano-42461

09/26/2023, 10:46 AM

also i checked the wf is pushed to s3

average-finland-92144

09/26/2023, 10:46 AM

do i need to create
flytesnacks-development
namespace?

no, this one is created as one of the

initialProjects

average-finland-92144

09/26/2023, 10:47 AM

The Pod has been restarted by K8s 9 times, there has to be something in logs. Can you also do a

kubectl describe po -n flyte flyte-binary-65fff8bf4b-n86kt

brash-piano-42461

09/26/2023, 10:48 AM

i updated the logs can u chk https://flyte-org.slack.com/archives/CP2HDHKE1/p1695725103484359?thread_ts=1695715859.488779&cid=CP2HDHKE1

brash-piano-42461

09/26/2023, 10:50 AM

what do u want to have a look in the describe thing

brash-piano-42461

09/26/2023, 10:50 AM

also we got some error becoz of ray which got resolved once we installed kuberay

brash-piano-42461

09/26/2023, 10:51 AM

i dont know whats the issue as everything looks fine

average-finland-92144

09/26/2023, 10:53 AM

can you verify that the SA has been annotated?

kubectl describe sa -n flyte flyte-binary

average-finland-92144

09/26/2023, 10:59 AM

so there are no Task Pods being created?

kubectl get po -A

brash-piano-42461

09/26/2023, 10:59 AM

brash-piano-42461

09/26/2023, 11:00 AM

Copy code

flyte         flyte-binary-65fff8bf4b-n86kt                   1/1     Running   9 (173m ago)   3h39m
kube-system   aws-cloudwatch-metrics-6hvr5                    1/1     Running   1 (21h ago)    21h
kube-system   aws-cloudwatch-metrics-cgb8v                    1/1     Running   0              21h
kube-system   aws-cloudwatch-metrics-zlb9n                    1/1     Running   0              21h
kube-system   aws-cluster-autoscaler-7878b6fcd5-mpjr2         1/1     Running   0              4h22m
kube-system   aws-load-balancer-controller-68d5864d5d-hjshh   1/1     Running   0              4h22m
kube-system   aws-load-balancer-controller-68d5864d5d-s74t2   1/1     Running   0              4h22m
kube-system   aws-node-5qgqz                                  1/1     Running   0              21h
kube-system   aws-node-8xpxn                                  1/1     Running   0              21h
kube-system   aws-node-cnsbt                                  1/1     Running   0              21h
kube-system   coredns-6bc4667bcc-g82mk                        1/1     Running   0              21h
kube-system   coredns-6bc4667bcc-w694l                        1/1     Running   0              21h
kube-system   external-dns-dbd54cbcb-x5vh4                    1/1     Running   0              21h
kube-system   kube-proxy-bmn98                                1/1     Running   0              21h
kube-system   kube-proxy-gwcsd                                1/1     Running   0              21h
kube-system   kube-proxy-ms8rr                                1/1     Running   0              21h
ray-system    kuberay-apiserver-bdb679dc-n9qg4                1/1     Running   0              3h31m
ray-system    kuberay-operator-dc7968898-qw8vq                1/1     Running   0              3h31m

brash-piano-42461

09/26/2023, 11:16 AM

@average-finland-92144?

average-finland-92144

09/26/2023, 3:04 PM

sorry, I had to drop (take the kids to school). Well, this is a strange behavior, can you check logs from the previous Pod execution?

kubectl logs -n flyte flyte-binary-65fff8bf4b-n86kt --previous

I've seen this behavior (

UNKNOWN

status) but every time it's been a K8s scheduling problem and I saw the Task Pods in

Pending

state. But in your env there's no Pod being created, so to me it looks more like a permissions issue somewhere, I think we can have a screen share session to debug every layer and isolate the problem

brash-piano-42461

09/26/2023, 3:25 PM

Ya sure

8 Views

Open in Slack

Previous Next