https://flyte.org
Join Slack
Hey community, I’m trying to deploy Flyte on my company’s AWS infrastructure following <this guide>....
# flyte-deployment
g
Hey community, I’m trying to deploy Flyte on my company’s AWS infrastructure following this guide. We’ve installed AWS ELB Controller. The pods seem to be healthy in the cluster but the ingress is not getting assigned any address for 
flyteconsole
due to the following error in the AWS ELB Controller logs:
Copy code
{
  "level": "error",
  "ts": 1659712067.8667307,
  "logger": "controller",
  "msg": "Reconciler error",
  "controller": "ingress",
  "name": "flyte",
  "namespace": "",
  "error": "InvalidParameter: 1 validation error(s) found.\n- minimum field value of 1, CreateTargetGroupInput.Port.\n"
}
Seems like the issue is https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/1695 due to the service that the ingress routes to having type 
clusterip
instead of 
nodeport
, yet the helm chart for flyteconsole define the service as type 
clusterip
. So I’m wondering if there’s something else that I’m not configuring correctly. If anyone else has run into something similar help would be very welcome! Thanks
g
I added:
Copy code
<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: 'ip'
and it works, the address shows up
But I struggled on something else now:)
g
thanks! That worked for me too! what is your issue?
g
I can access the flyte console with 
elb adresss/console
, but when I execute 
flytectl create project
, it returns:
Copy code
{
  "json": {
    "src": "main.go:13"
  },
  "level": "error",
  "msg": "rpc error: code = Unavailable desc = connection closed",
  "ts": "2022-08-08T17:26:10+02:00"
}
Would you please let me know if it works for you? (Either the thread above or the flyte guide)
f
@gray-lifeguard-89900 / @glamorous-artist-35470 always the freaking ingress 🙂
thank you for helping each other - highly appreciate the camaraderie
@glamorous-artist-35470 are you running the alb ingress controller?
@gray-lifeguard-89900 / @glamorous-artist-35470 so flytectl uses grpc to communicate. While Flyteconsole uses http (REST). thus console works and flytectl does not. you have to enable TCP port. But if you use alb, It supports grpc
g
@freezing-airport-6809 I’m using ALB, but am having trouble accessing the flyteconsole UI 😄
f
ohh you are
for all the folks who struggle with Ingress and we help and they get moving and then forget about it. I think once you folks get it setup, we would love for you to help us update the docs - what made it work, where you got stuck
g
@freezing-airport-6809 Thanks for your reply!
@glamorous-artist-35470 If your pod is up and running: I think you can also check if internet gateway is attached to your subnets of your ALB, if your subnets include one that is not a public one you may not access the console…
f
cc @great-school-54368 if you know what could be the problem here
g
I got FlyteConsole UI up and running yay! @gray-lifeguard-89900 I think I’m running into the same issue when I try to create 
flytectl create
I get 
Error: rpc error: code = Unavailable desc = connection closed
. I tried changing HTTP2 to GRPC for the grpc ingress: 
<http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
but it didn’t work. See: https://github.com/flyteorg/flyte/pull/2702/files#diff-33b4463f6057591a533425d1f947752711a81da1952ff745ed9fae049e155995 @freezing-airport-6809 happy to update docs once I get this up and running!
g
Exactly, I am struggling with this for the whole day hehe
g
Have you made any progress on this issue? I have tried setting 
<http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: HTTP1
and disabling 
albSSLRedirect: false
in the config. Now when I try to access flytectl, I get 
Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: "<cdn for certificate domain>" certificate name does not match input
. Trying to make inroads using this workaround
f
@glamorous-artist-35470 / @gray-lifeguard-89900 sorry for the delayed response
cc @great-school-54368 can you please help here?
g
@glamorous-artist-35470 Can you try flytectl with insecureSkipVerify ?
Copy code
flytectl get projects --admin.insecureSkipVerify --logger.level=10
g
Sorry to borrow the thread but this is what I got when running command above:
Copy code
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-08-09T09:31:24+02:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-08-09T09:31:24+02:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-08-09T09:31:24+02:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2022-08-09T09:31:24+02:00"}
{"json":{"src":"client.go:183"},"level":"error","msg":"failed to initialize token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unavailable desc = connection closed","ts":"2022-08-09T09:31:25+02:00"}
{"json":{"src":"client.go:188"},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-08-09T09:31:25+02:00"}
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-08-09T09:31:25+02:00"}
Error: rpc error: code = Unavailable desc = connection closed
{"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = Unavailable desc = connection closed","ts":"2022-08-09T09:31:25+02:00"}
@great-school-54368
g
@gray-lifeguard-89900 if you have time then can we debug it on google meet ?
g
That would be perfect
Got really handy and prompt help from @great-school-54368. Big kudos to you! And this is how we solve the current AWS(EKS) manual setup issue if you are facing the same: 1. Use this version of values-eks.yaml when you installing flyte with Helm, the current master version creates some issue and people are investigating 2. Comment out 
affinity
section for each pods if you have pending state pods, you can ignore this if you don’t have pending state pods because of affinity 3. In the Connecting to Flyte Section if you are using a self-signed certificates, in the 
~/.flyte/config.yaml
, write 
insecure: false
instead of 
true
since you have a certificate anyways so it’s not insecure. Leave 
insecureSkipVerify: true
then your certificate won’t be verified. (Should not apply to production as mentioned in setup)
🙌 5
❤️ 2
👍 1
@glamorous-artist-35470
g
@gray-lifeguard-89900 Thanks for updating the thread. @glamorous-artist-35470 You can skip 2nd step if you are not facing any issue, After helm deployment please validate all the pods status by running 
kubectl get pod -n flyte
f
@gray-lifeguard-89900 want to propose a change for docs
g
g
Thanks for the reply guys! I am getting 
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-08-09T10:33:18-04:00"}
Error: rpc error: code = Unknown desc = : HTTP status code 464; transport: missing content-type field
{"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = Unknown desc = : HTTP status code 464; transport: missing content-type field","ts":"2022-08-09T10:33:18-04:00"}
I’m using ALB. I have 
<http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
and have tried the config settings that @gray-lifeguard-89900 suggested above I’m running: 
flytectl get projects --admin.insecureSkipVerify --logger.level=10
g
@glamorous-artist-35470 do you have time for debugging the issue on google meet ?
1577 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by