Hello guys I need some help We are doing some testing with f

Question

Hello guys I need some help We are doing some testing with flyte in the AWS environment with ALB But I m having trouble trying to run the example code ```import typing import pandas as pd import numpy as np from flytekit import task workflow task def generate normal df n int mean float sigma float gt pd DataFrame return pd DataFrame numbers np random normal mean sigma size=n task def compute stats df pd DataFrame gt typing Tuple float float return float df numbers mean float df numbers std workflow def wf n int = 200 mean float = 0 0 sigma float = 1 0 gt typing Tuple float float return compute stats df=generate normal df n=n mean=mean sigma=sigma ``` When I run the code using the local cluster using flytectl demo start k3s it works But in production environment using EKS and ALB does not work ```pyflyte run remote example py wf n 500 mean 42 sigma 2 SIGINT 2 10055 17 06 22 asctime 2022 05 06 17 06 31 287 name flytekit cli levelname ERROR message Non auth RPC error lt InactiveRpcError of RPC that terminated with n tstatus = StatusCode UNAVAILABLE n tdetails = failed to connect to all addresses n tdebug error string = created 1651867591 287595583 description Failed to pick subchannel file src core ext filters client channel client channel cc file line 3128 referenced errors created 1651867591 287594125 description failed to connect to all addresses file src core lib transport error utils cc file line 163 grpc status 14 n gt sleeping 200ms and retrying asctime 2022 05 06 17 06 32 274 name flytekit cli levelname ERROR message Non auth RPC error lt InactiveRpcError of RPC that terminated with n tstatus = StatusCode UNAVAILABLE n tdetails = failed to connect to all addresses n tdebug error string = created 1651867592 274550785 description Failed to pick subchannel file src core ext filters client channel client channel cc file line 3128 referenced errors created 1651867592 274550285 description failed to connect to all addresses file src core lib transport error utils cc file line 163 grpc status 14 n gt sleeping 400ms and retrying Traceback most recent call last File usr local bin pyflyte line 8 in lt module gt sys exit main File home lalmeida local lib python3 8 site packages click core py line 1128 in call return self main args kwargs File home lalmeida local lib python3 8 site packages click core py line 1053 in main rv = self invoke ctx File home lalmeida local lib python3 8 site packages click core py line 1659 in invoke return process result sub ctx command invoke sub ctx File home lalmeida local lib python3 8 site packages click core py line 1659 in invoke return process result sub ctx command invoke sub ctx File home lalmeida local lib python3 8 site packages click core py line 1659 in invoke return process result sub ctx command invoke sub ctx File home lalmeida local lib python3 8 site packages click core py line 1395 in invoke return ctx invoke self callback ctx params File home lalmeida local lib python3 8 site packages click core py line 754 in invoke return callback args kwargs File usr local lib python3 8 dist packages flytekit clis sdk in container run py line 471 in run wf = remote register script File usr local lib python3 8 dist packages flytekit remote remote py line 536 in register script upload location md5 bytes = fast register single script File usr local lib python3 8 dist packages flytekit tools script mode py line 117 in fast register single script upload location = create upload location fn content md5=md5 File usr local lib python3 8 dist packages flytekit clients friendly py line 998 in get upload signed url return super SynchronousFlyteClient self create upload location File usr local lib python3 8 dist packages flytekit clients raw py line 40 in handler return fn args kwargs File usr local lib python3 8 dist packages flytekit clients raw py line 834 in create upload location return self dataproxy stub CreateUploadLocation create upload location request metadata=self metadata File usr local lib python3 8 dist packages grpc channel py line 946 in call return end unary response blocking state call False None File usr local lib python3 8 dist packages grpc channel py line 849 in end unary response blocking raise InactiveRpcError state grpc channel InactiveRpcError lt InactiveRpcError of RPC that terminated with status = StatusCode UNAVAILABLE details = failed to connect to all addresses debug error string = created 1651867592 675517306 description Failed to pick subchannel file src core ext filters client channel client channel cc file line 3128 referenced errors created 1651867592 675516568 description failed to connect to all addresses file src core lib transport error utils cc file line 163 grpc status 14 ``` Do you have any ideas I m using helm with values eks yaml

Yee · Answer

unavailable means it can t even find the endpoint i believe

Yee · Answer

can you paste your `~ flyte config yaml` and `~ flyte config` file if they exist and also `env | grep i flyte`

Yee · Answer

and the flyte UI works right

Yee · Answer

who is doing your ingress

Yee · Answer

is it the alb ingress controller

Yee · Answer

or is nginx involved

Leonardo Almeida · Answer

config yaml ```admin For GRPC endpoints you might want to use dns flyte myexample com endpoint dns internal k8s flyte a58b2df7fa 1695464265 us east 1 elb amazonaws com authType Pkce insecure true only required when using insecure ingress Secure ingress may cause an unavailable desc error to true option ```

Leonardo Almeida · Answer

Yes flyte UI works

Yee · Answer

insecure probably should be false right

Leonardo Almeida · Answer

It s a alb ingress controller

Yee · Answer

i thought alb came with certs

Yee · Answer

and when you do kubectl n flyte get ingress that s the same address

Yee · Answer

the only other thing i can think of is that sometimes the alb isn t setup to to use HTTP2 Preferred on some setting or maybe that was an nginx issue

Yee · Answer

like alpn or something

Leonardo Almeida · Answer

yes ``` kubectl get ingress n flyte 10067 17 46 32 NAME CLASS HOSTS ADDRESS PORTS AGE flyte core lt none gt 80 4h49m flyte core grpc lt none gt 80 4h49m```

Yee · Answer

can you take a look at ingress logs

Yee · Answer

do you have time for a quick screenshare in ten mins

Leonardo Almeida · Answer

I m sorry I m very busy But now I have another error ```grpc channel InactiveRpcError lt InactiveRpcError of RPC that terminated with status = StatusCode UNKNOWN details = failed to create a signed url Error WebIdentityErr failed to retrieve credentials caused by ValidationError Request ARN is invalid status code 400 request id 6b14cd70 5a90 43a7 a8f2 e03997c937db debug error string = created 1652131218 092115614 description Error received from peer ipv4 10 0 28 93 443 file src core lib surface call cc file line 903 grpc message failed to create a signed url Error WebIdentityErr failed to retrieve credentials ncaused by ValidationError Request ARN is invalid n tstatus code 400 request id 6b14cd70 5a90 43a7 a8f2 e03997c937db grpc status 2 gt ```

Yee · Answer

what was the command that gave this error

Yee · Answer

still down to have a quick chat if you are free

Leonardo Almeida · Answer

Now it worked In the helm in the accountNumber field it needed single quotes and I wasn t passing it So the service account was coming in an incorrect format and generating the error

Leonardo Almeida · Answer

thank you very much