https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
a

Ahmed Laadraoui

03/16/2023, 1:40 PM
Hi <!channel>, I hope you all are doing great. I am new to FLyte and I want to learn it as I really see a big potential in it. I just deployed Flyte in my AWS account using Opta. Now I am trying to create a project and push some workflows and run them. Any documentation link, articles or explanations will be very helpful to help me establish a connection with my deployed Flyte instance and be able to process tasks and workflows remotely. Many Tanks Ahmed
d

David Espejo (he/him)

03/16/2023, 1:48 PM
Hi @Ahmed Laadraoui and welcome. I'm glad you have Flyte already deployed. You could go to the Getting started guide (assuming you haven't done so) and check the Creating a workflow section Once created you could then run it against your Flyte on EKS 
pyflyte run --remote example.py training_workflow \
--hyperparameters '{"C": 0.1}'
btw you can skip the 
flytectl demo start
part because you already have a Flyte backend deployed Let us know here about your progress, We try to be as responsive as possible, so really at channel is not necessary 🙂
a

Ahmed Laadraoui

03/16/2023, 1:50 PM
Hi David, Thank you for your quick response 🙂
I already did the get started and I run some flows locally
d

David Espejo (he/him)

03/16/2023, 1:51 PM
great! there are also Tutorials for different use cases: https://docs.flyte.org/projects/cookbook/en/latest/tutorials.html How's been your experience so far with Flyte?
a

Ahmed Laadraoui

03/16/2023, 1:52 PM
It was very difficult in term of documentation, there are lot of information but the process is not very clear
not only for me but for all the team that is working on it currently
d

David Espejo (he/him)

03/16/2023, 1:52 PM
you mean, for the deployment on EKS?
a

Ahmed Laadraoui

03/16/2023, 1:54 PM
there was some issue that took me time, for example sometimes the local instance stops running, and I got a docker error saying that it’s already used when I try to restart it (talking about the demo start), the solution was to shut it down. this is a very important note that should be put in the first getting started for example
Actually the deployment was straight forward with Opta
d

David Espejo (he/him)

03/16/2023, 1:57 PM
I got a docker error saying that it’s already used when I try to restart it
like, the port is already in use?
a

Ahmed Laadraoui

03/16/2023, 1:57 PM
Now My Flyte instance is live. I want to push some workflows there. I added this config file in the root directory of the project:
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///<my-domain.xyz>
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
I suppose this will connect my flytectl with the deployed version
When now I am trying to create a new project I am getting a timout error
d

David Espejo (he/him)

03/16/2023, 1:58 PM
ok, that config is to connect to your Flyte instance on EKS right?
a

Ahmed Laadraoui

03/16/2023, 1:59 PM
Yes
d

David Espejo (he/him)

03/16/2023, 1:59 PM
what Helm chart you used to deploy, was it 
flyte-binary
?
oh, you used Opta
a

Ahmed Laadraoui

03/16/2023, 2:01 PM
Yes 😅
d

David Espejo (he/him)

03/16/2023, 2:02 PM
are you using self signed certs or full SSL?
a

Ahmed Laadraoui

03/16/2023, 2:04 PM
What do you mean by that sir? For AWS connexion?
d

David Espejo (he/him)

03/16/2023, 2:06 PM
yes, from the AWS side, does that 
<http://my-domain.xyz|my-domain.xyz>
has a corresponding certificate created and imported?
a

Ahmed Laadraoui

03/16/2023, 2:06 PM
No I didn’t create any certificate
I just added the records in my domain provider and let Opta do the rest
it’s actually working
it’s not my-domain.xyz
I just put it there for the example
I can send you the real domain
d

David Espejo (he/him)

03/16/2023, 2:07 PM
gotcha 🙂
np
so, what happens when you go to http://your-domain:8088/console ?
a

Ahmed Laadraoui

03/16/2023, 2:09 PM
in I have the flyte console running http://my-domain.xyz/console
d

David Espejo (he/him)

03/16/2023, 2:10 PM
gotcha, I've seen timeout errors in my flyte backend on EKS when I don't specify a port in the `config.yaml`file. In this case for a non SSL grpc connection: 
endpoint: dns:///my-domain:8089
I mean, timeouts when trying to submit workflows
a

Ahmed Laadraoui

03/16/2023, 2:11 PM
with no .xyz?
not like that? 
dns:///my-domain.xyz:8089
trying it now
d

David Espejo (he/him)

03/16/2023, 2:12 PM
yes, whatever your domain is 😛
a

Ahmed Laadraoui

03/16/2023, 2:12 PM
Sure 1 sec 🙂
still getting timout
I am running a command to create a project: 
flytectl create project --file project.yaml
d

David Espejo (he/him)

03/16/2023, 2:14 PM
ok, how are you connecting to the EKS cluster. are you port-forwarding or using Ingress?
a

Alexander Kim

03/16/2023, 2:15 PM
There are almost 2k people on this channel; please be considerate to others and avoid using “@ channel” in the future. Thank you!
a

Ahmed Laadraoui

03/16/2023, 2:16 PM
None of these ^^, I have my aws credentials in the environments that’s it. I didn’t find any guide to connect to EKS, I don’t know what pod or depl or srv to connect to
as I supposed that this config will setup the connexion to the remote server
d

David Espejo (he/him)

03/16/2023, 2:18 PM
check which service was created in your namespace (typically 
flyte
) 
kubectl get svc -n flyte
a

Ahmed Laadraoui

03/16/2023, 2:18 PM
Screen Shot 2023-03-16 at 3.18.33 PM.png
d

David Espejo (he/him)

03/16/2023, 2:25 PM
ok, definitely not 
flyte-binary
🙂 alright, so as this is a non SSL connection, you could 1. Use the :80 port in your 
config.yaml
 2. Start a port-forwarding session to test connection, something like 
kubectl -n flyte port-forward service/flyteadmin 8089:80
a

Ahmed Laadraoui

03/16/2023, 2:28 PM
done
I guess if I forward the port I can use my localhost ?
in the config file
d

David Espejo (he/him)

03/16/2023, 2:31 PM
you could yes, sometimes it causes errors as python gRPC is not very friendly to self signed certificates (that's why I asked) but you could try
a

Ahmed Laadraoui

03/16/2023, 2:32 PM
I just run this 
flytectl create project --file project.yaml
Error: Connection Info: [Endpoint: dns:///localhost:8089, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
d

David Espejo (he/him)

03/16/2023, 2:32 PM
n if you have a self signed cert already generated for that domain name, the name has to match to what you have on 
config
otherwise it will throw weird SSL errors
a

Ahmed Laadraoui

03/16/2023, 2:32 PM
I see the kubectl forward was excited: 
Handling connection for 8089
d

David Espejo (he/him)

03/16/2023, 2:33 PM
could you increase the logging level? 
logger:
show-source: true
level: 6
a

Ahmed Laadraoui

03/16/2023, 2:34 PM
Screen Shot 2023-03-16 at 3.34.03 PM.png
I guess some update handlers are needed?
Full Log:
Copy code
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2023-03-16T15:33:38+01:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2023-03-16T15:33:38+01:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2023-03-16T15:33:38+01:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2023-03-16T15:33:38+01:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [console] updated. No update handler registered.","ts":"2023-03-16T15:33:38+01:00"}
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2023-03-16T15:33:38+01:00"}
{"json":{"src":"auth_interceptor.go:67"},"level":"debug","msg":"Request failed due to [rpc error: code = Unavailable desc = connection closed before server preface received]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2023-03-16T15:34:19+01:00"}
Error: Connection Info: [Endpoint: dns:///localhost:8089, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
{"json":{"src":"main.go:13"},"level":"error","msg":"Connection Info: [Endpoint: dns:///localhost:8089, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received","ts":"2023-03-16T15:34:19+01:00"}
this is my config: 
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///localhost:8089
authType: Pkce
insecure: true
logger:
show-source: true
level: 6
d

David Espejo (he/him)

03/16/2023, 2:38 PM
what about using your domain instead of localhost?
a

Ahmed Laadraoui

03/16/2023, 2:38 PM
without port?
d

David Espejo (he/him)

03/16/2023, 2:38 PM
with the port
a

Ahmed Laadraoui

03/16/2023, 2:39 PM
or with :80
Different Logs now
Copy code
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2023-03-16T15:39:19+01:00"}
{"json":{"src":"auth_interceptor.go:67"},"level":"debug","msg":"Request failed due to [rpc error: code = Unavailable desc = connection closed before server preface received]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2023-03-16T15:39:40+01:00"}
Error: Connection Info: [Endpoint: dns:///opentick.xyz:80, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
{"json":{"src":"main.go:13"},"level":"error","msg":"Connection Info: [Endpoint: dns:///opentick.xyz:80, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received","ts":"2023-03-16T15:39:40+01:00"}
d

David Espejo (he/him)

03/16/2023, 2:40 PM
it should be 80. Your service doesn't expose 8089
and I guess your domain resolves to some public address right? Anyways, the port-forward is a temporary way to test connection so probably the last change I'd ask you would be to use again 
localhost:80
and if it doesn't work, we'll need more information on your setup (certificates, ingress or not, etc)
a

Ahmed Laadraoui

03/16/2023, 2:46 PM
All I did is: • Deploy Flyte on my AWS account using Opta => FLyte console is now available in my domain • Changed the flyte config file to add the domain where the consol is working
What are the supposed steps? after deploying using Opta?
Still getting error
Copy code
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2023-03-16T15:44:30+01:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2023-03-16T15:44:30+01:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2023-03-16T15:44:30+01:00"}
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2023-03-16T15:44:30+01:00"}
{"json":{"src":"auth_interceptor.go:67"},"level":"debug","msg":"Request failed due to [rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:80: connect: connection refused\"]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2023-03-16T15:45:10+01:00"}
Error: Connection Info: [Endpoint: dns:///localhost:80, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:80: connect: connection refused"
{"json":{"src":"main.go:13"},"level":"error","msg":"Connection Info: [Endpoint: dns:///localhost:80, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:80: connect: connection refused\"","ts":"2023-03-16T15:45:10+01:00"}
d

David Espejo (he/him)

03/16/2023, 2:51 PM
isn't the 80 port already in use in your machine?
this is a different error: 
connection refused
also, btw, your port-forward session should change: 
kubectl -n flyte port-forward service/flyteadmin 80:80
a

Ahmed Laadraoui

03/16/2023, 2:56 PM
I am unable to forward to 80:
Copy code
Unable to listen on port 80: Listeners failed to create with the following errors: [unable to create listener: Error listen tcp4 127.0.0.1:80: bind: permission denied unable to create listener: Error listen tcp6 [::1]:80: bind: permission denied]
error: unable to listen on any of the requested ports: [{80 8088}]
Actually I didn’t add any additional config to Opta, it automatically deployed to 80 as you see
d

David Espejo (he/him)

03/16/2023, 3:01 PM
right, but seems that your local :80 port is busy
can you share your opta config?
a

Ahmed Laadraoui

03/16/2023, 3:02 PM
I used these
Btw when running the first Opta run to the .env, the last stem of the runing wasn’t successful:
But the rest was
I got the error when running the whole thing, this screen shot shows the result when I retried it, as you see the Terraform plan only found one item pending witch is 
helm_release.metrics_server
Can this cause the issue?
@David Espejo (he/him) Any comment sir?
d

David Espejo (he/him)

03/16/2023, 7:11 PM
sorry, was in meetings. So, Opta should have created an Ingress resource and you should be using it to connect. Can you confirm there's an Ingress created and the status of the backing Service?
a

Ahmed Laadraoui

03/16/2023, 7:51 PM
Hi David
thank you for your response, no worries 🙂
Yes there are ingresses
I just created the project
Screen Shot 2023-03-16 at 8.58.38 PM.png
I used port 81 and it works
d

David Espejo (he/him)

03/16/2023, 7:59 PM
hm, so one could say that, similar to single binary deployment, 
81
in this case would be the port for non SSL connections, interesting
a

Ahmed Laadraoui

03/16/2023, 8:00 PM
Yeah
I tested the 87 witch throws a crazy error and crashes the forward
then the 81 🙂
I finally created the project, now I need to push some workflows
d

David Espejo (he/him)

03/16/2023, 9:10 PM
awesome @Ahmed Laadraoui!
a

Ahmed Laadraoui

03/16/2023, 10:42 PM
Thank you 🙂
can you please tell me why the minimum task duration is around 15s even if I only return a small number in the task function?
16 Views
Powered by