https://flyte.org logo
Powered by
a

allen

08/08/2022, 2:45 PM
Hey community, I’m trying to deploy Flyte on my company’s AWS infrastructure following this guide. We’ve installed AWS ELB Controller. The pods seem to be healthy in the cluster but the ingress is not getting assigned any address for 
flyteconsole
due to the following error in the AWS ELB Controller logs:
Copy code
{
  "level": "error",
  "ts": 1659712067.8667307,
  "logger": "controller",
  "msg": "Reconciler error",
  "controller": "ingress",
  "name": "flyte",
  "namespace": "",
  "error": "InvalidParameter: 1 validation error(s) found.\n- minimum field value of 1, CreateTargetGroupInput.Port.\n"
}
Seems like the issue is https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/1695 due to the service that the ingress routes to having type 
clusterip
instead of 
nodeport
, yet the helm chart for flyteconsole define the service as type 
clusterip
. So I’m wondering if there’s something else that I’m not configuring correctly. If anyone else has run into something similar help would be very welcome! Thanks
j

Jacob Wang

08/08/2022, 3:10 PM
I added:
Copy code
<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: 'ip'
and it works, the address shows up
But I struggled on something else now:)
a

allen

08/08/2022, 3:27 PM
thanks! That worked for me too! what is your issue?
j

Jacob Wang

08/08/2022, 3:28 PM
I can access the flyte console with 
elb adresss/console
, but when I execute 
flytectl create project
, it returns:
Copy code
{
  "json": {
    "src": "main.go:13"
  },
  "level": "error",
  "msg": "rpc error: code = Unavailable desc = connection closed",
  "ts": "2022-08-08T17:26:10+02:00"
}
Would you please let me know if it works for you? (Either the thread above or the flyte guide)
k

Ketan (kumare3)

08/08/2022, 3:40 PM
@Jacob Wang / @allen always the freaking ingress 🙂
thank you for helping each other - highly appreciate the camaraderie
@allen are you running the alb ingress controller?
@Jacob Wang / @allen so flytectl uses grpc to communicate. While Flyteconsole uses http (REST). thus console works and flytectl does not. you have to enable TCP port. But if you use alb, It supports grpc
a

allen

08/08/2022, 3:46 PM
@Ketan (kumare3) I’m using ALB, but am having trouble accessing the flyteconsole UI 😄
k

Ketan (kumare3)

08/08/2022, 3:47 PM
ohh you are
for all the folks who struggle with Ingress and we help and they get moving and then forget about it. I think once you folks get it setup, we would love for you to help us update the docs - what made it work, where you got stuck
j

Jacob Wang

08/08/2022, 3:50 PM
@Ketan (kumare3) Thanks for your reply!
@allen If your pod is up and running: I think you can also check if internet gateway is attached to your subnets of your ALB, if your subnets include one that is not a public one you may not access the console…
k

Ketan (kumare3)

08/08/2022, 3:58 PM
cc @Yuvraj if you know what could be the problem here
a

allen

08/08/2022, 4:38 PM
I got FlyteConsole UI up and running yay! @Jacob Wang I think I’m running into the same issue when I try to create 
flytectl create
I get 
Error: rpc error: code = Unavailable desc = connection closed
. I tried changing HTTP2 to GRPC for the grpc ingress: 
<http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
but it didn’t work. See: https://github.com/flyteorg/flyte/pull/2702/files#diff-33b4463f6057591a533425d1f947752711a81da1952ff745ed9fae049e155995 @Ketan (kumare3) happy to update docs once I get this up and running!
j

Jacob Wang

08/08/2022, 4:40 PM
Exactly, I am struggling with this for the whole day hehe
a

allen

08/08/2022, 8:05 PM
Have you made any progress on this issue? I have tried setting 
<http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: HTTP1
and disabling 
albSSLRedirect: false
in the config. Now when I try to access flytectl, I get 
Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: "<cdn for certificate domain>" certificate name does not match input
. Trying to make inroads using this workaround
k

Ketan (kumare3)

08/09/2022, 5:13 AM
@allen / @Jacob Wang sorry for the delayed response
cc @Yuvraj can you please help here?
y

Yuvraj

08/09/2022, 5:33 AM
@allen Can you try flytectl with insecureSkipVerify ?
Copy code
flytectl get projects --admin.insecureSkipVerify --logger.level=10
j

Jacob Wang

08/09/2022, 7:33 AM
Sorry to borrow the thread but this is what I got when running command above:
Copy code
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-08-09T09:31:24+02:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-08-09T09:31:24+02:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-08-09T09:31:24+02:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2022-08-09T09:31:24+02:00"}
{"json":{"src":"client.go:183"},"level":"error","msg":"failed to initialize token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unavailable desc = connection closed","ts":"2022-08-09T09:31:25+02:00"}
{"json":{"src":"client.go:188"},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-08-09T09:31:25+02:00"}
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-08-09T09:31:25+02:00"}
Error: rpc error: code = Unavailable desc = connection closed
{"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = Unavailable desc = connection closed","ts":"2022-08-09T09:31:25+02:00"}
@Yuvraj
y

Yuvraj

08/09/2022, 7:34 AM
@Jacob Wang if you have time then can we debug it on google meet ?
j

Jacob Wang

08/09/2022, 7:34 AM
That would be perfect
Got really handy and prompt help from @Yuvraj. Big kudos to you! And this is how we solve the current AWS(EKS) manual setup issue if you are facing the same: 1. Use this version of values-eks.yaml when you installing flyte with Helm, the current master version creates some issue and people are investigating 2. Comment out 
affinity
section for each pods if you have pending state pods, you can ignore this if you don’t have pending state pods because of affinity 3. In the Connecting to Flyte Section if you are using a self-signed certificates, in the 
~/.flyte/config.yaml
, write 
insecure: false
instead of 
true
since you have a certificate anyways so it’s not insecure. Leave 
insecureSkipVerify: true
then your certificate won’t be verified. (Should not apply to production as mentioned in setup)
🙌 5
❤️ 2
👍 1
@allen
y

Yuvraj

08/09/2022, 8:37 AM
@Jacob Wang Thanks for updating the thread. @allen You can skip 2nd step if you are not facing any issue, After helm deployment please validate all the pods status by running 
kubectl get pod -n flyte
k

Ketan (kumare3)

08/09/2022, 1:43 PM
@Jacob Wang want to propose a change for docs
j

Jacob Wang

08/09/2022, 2:08 PM
a

allen

08/09/2022, 2:34 PM
Thanks for the reply guys! I am getting 
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-08-09T10:33:18-04:00"}
Error: rpc error: code = Unknown desc = : HTTP status code 464; transport: missing content-type field
{"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = Unknown desc = : HTTP status code 464; transport: missing content-type field","ts":"2022-08-09T10:33:18-04:00"}
I’m using ALB. I have 
<http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
and have tried the config settings that @Jacob Wang suggested above I’m running: 
flytectl get projects --admin.insecureSkipVerify --logger.level=10
y

Yuvraj

08/09/2022, 3:25 PM
@allen do you have time for debugging the issue on google meet ?
997 Views
Powered by