https://flyte.org logo
Join Slack
Powered by
Hi <!channel>, I hope you all are doing great. I a...
# ask-the-community
a
Hi <!channel>, I hope you all are doing great. I am new to FLyte and I want to learn it as I really see a big potential in it. I just deployed Flyte in my AWS account using Opta. Now I am trying to create a project and push some workflows and run them. Any documentation link, articles or explanations will be very helpful to help me establish a connection with my deployed Flyte instance and be able to process tasks and workflows remotely. Many Tanks Ahmed
d
Hi @Ahmed Laadraoui and welcome. I'm glad you have Flyte already deployed. You could go to the Getting started guide (assuming you haven't done so) and check the Creating a workflow section Once created you could then run it against your Flyte on EKS 
pyflyte run --remote example.py training_workflow \
--hyperparameters '{"C": 0.1}'
btw you can skip the 
flytectl demo start
part because you already have a Flyte backend deployed Let us know here about your progress, We try to be as responsive as possible, so really at channel is not necessary šŸ™‚
a
Hi David, Thank you for your quick response šŸ™‚
I already did the get started and I run some flows locally
d
great! there are also Tutorials for different use cases: https://docs.flyte.org/projects/cookbook/en/latest/tutorials.html How's been your experience so far with Flyte?
a
It was very difficult in term of documentation, there are lot of information but the process is not very clear
not only for me but for all the team that is working on it currently
d
you mean, for the deployment on EKS?
a
there was some issue that took me time, for example sometimes the local instance stops running, and I got a docker error saying that itā€™s already used when I try to restart it (talking about the demo start), the solution was to shut it down. this is a very important note that should be put in the first getting started for example
Actually the deployment was straight forward with Opta
d
I got a docker error saying that itā€™s already used when I try to restart it
like, the port is already in use?
a
Now My Flyte instance is live. I want to push some workflows there. I added this config file in the root directory of the project:
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///<my-domain.xyz>
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
I suppose this will connect my flytectl with the deployed version
When now I am trying to create a new project I am getting a timout error
d
ok, that config is to connect to your Flyte instance on EKS right?
a
Yes
d
what Helm chart you used to deploy, was it 
flyte-binary
?
oh, you used Opta
a
Yes šŸ˜…
d
are you using self signed certs or full SSL?
a
What do you mean by that sir? For AWS connexion?
d
yes, from the AWS side, does that 
<http://my-domain.xyz|my-domain.xyz>
has a corresponding certificate created and imported?
a
No I didnā€™t create any certificate
I just added the records in my domain provider and let Opta do the rest
itā€™s actually working
itā€™s not my-domain.xyz
I just put it there for the example
I can send you the real domain
d
gotcha šŸ™‚
np
so, what happens when you go to http://your-domain:8088/console ?
a
in I have the flyte console running http://my-domain.xyz/console
d
gotcha, I've seen timeout errors in my flyte backend on EKS when I don't specify a port in the `config.yaml`file. In this case for a non SSL grpc connection: 
endpoint: dns:///my-domain:8089
I mean, timeouts when trying to submit workflows
a
with no .xyz?
not like that? 
dns:///my-domain.xyz:8089
trying it now
d
yes, whatever your domain is šŸ˜›
a
Sure 1 sec šŸ™‚
still getting timout
I am running a command to create a project: 
flytectl create project --file project.yaml
d
ok, how are you connecting to the EKS cluster. are you port-forwarding or using Ingress?
a
There are almost 2k people on this channel; please be considerate to others and avoid using ā€œ@ channelā€ in the future. Thank you!
a
None of these ^^, I have my aws credentials in the environments thatā€™s it. I didnā€™t find any guide to connect to EKS, I donā€™t know what pod or depl or srv to connect to
as I supposed that this config will setup the connexion to the remote server
d
check which service was created in your namespace (typically 
flyte
) 
kubectl get svc -n flyte
a
Screen Shot 2023-03-16 at 3.18.33 PM.png
d
ok, definitely not 
flyte-binary
šŸ™‚ alright, so as this is a non SSL connection, you could 1. Use the :80 port in your 
config.yaml
 2. Start a port-forwarding session to test connection, something like 
kubectl -n flyte port-forward service/flyteadmin 8089:80
a
done
I guess if I forward the port I can use my localhost ?
in the config file
d
you could yes, sometimes it causes errors as python gRPC is not very friendly to self signed certificates (that's why I asked) but you could try
a
I just run this 
flytectl create project --file project.yaml
Error: Connection Info: [Endpoint: dns:///localhost:8089, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
d
n if you have a self signed cert already generated for that domain name, the name has to match to what you have on 
config
otherwise it will throw weird SSL errors
a
I see the kubectl forward was excited: 
Handling connection for 8089
d
could you increase the logging level? 
logger:
show-source: true
level: 6
a
Screen Shot 2023-03-16 at 3.34.03 PM.png
I guess some update handlers are needed?
Full Log:
Copy code
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2023-03-16T15:33:38+01:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2023-03-16T15:33:38+01:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2023-03-16T15:33:38+01:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2023-03-16T15:33:38+01:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [console] updated. No update handler registered.","ts":"2023-03-16T15:33:38+01:00"}
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2023-03-16T15:33:38+01:00"}
{"json":{"src":"auth_interceptor.go:67"},"level":"debug","msg":"Request failed due to [rpc error: code = Unavailable desc = connection closed before server preface received]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2023-03-16T15:34:19+01:00"}
Error: Connection Info: [Endpoint: dns:///localhost:8089, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
{"json":{"src":"main.go:13"},"level":"error","msg":"Connection Info: [Endpoint: dns:///localhost:8089, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received","ts":"2023-03-16T15:34:19+01:00"}
this is my config: 
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///localhost:8089
authType: Pkce
insecure: true
logger:
show-source: true
level: 6
d
what about using your domain instead of localhost?
a
without port?
d
with the port
a
or with :80
Different Logs now
Copy code
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2023-03-16T15:39:19+01:00"}
{"json":{"src":"auth_interceptor.go:67"},"level":"debug","msg":"Request failed due to [rpc error: code = Unavailable desc = connection closed before server preface received]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2023-03-16T15:39:40+01:00"}
Error: Connection Info: [Endpoint: dns:///opentick.xyz:80, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
{"json":{"src":"main.go:13"},"level":"error","msg":"Connection Info: [Endpoint: dns:///opentick.xyz:80, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received","ts":"2023-03-16T15:39:40+01:00"}
d
it should be 80. Your service doesn't expose 8089
and I guess your domain resolves to some public address right? Anyways, the port-forward is a temporary way to test connection so probably the last change I'd ask you would be to use again 
localhost:80
and if it doesn't work, we'll need more information on your setup (certificates, ingress or not, etc)
a
All I did is: ā€¢ Deploy Flyte on my AWS account using Opta => FLyte console is now available in my domain ā€¢ Changed the flyte config file to add the domain where the consol is working
What are the supposed steps? after deploying using Opta?
Still getting error
Copy code
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2023-03-16T15:44:30+01:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2023-03-16T15:44:30+01:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2023-03-16T15:44:30+01:00"}
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2023-03-16T15:44:30+01:00"}
{"json":{"src":"auth_interceptor.go:67"},"level":"debug","msg":"Request failed due to [rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:80: connect: connection refused\"]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2023-03-16T15:45:10+01:00"}
Error: Connection Info: [Endpoint: dns:///localhost:80, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:80: connect: connection refused"
{"json":{"src":"main.go:13"},"level":"error","msg":"Connection Info: [Endpoint: dns:///localhost:80, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:80: connect: connection refused\"","ts":"2023-03-16T15:45:10+01:00"}
d
isn't the 80 port already in use in your machine?
this is a different error: 
connection refused
also, btw, your port-forward session should change: 
kubectl -n flyte port-forward service/flyteadmin 80:80
a
I am unable to forward to 80:
Copy code
Unable to listen on port 80: Listeners failed to create with the following errors: [unable to create listener: Error listen tcp4 127.0.0.1:80: bind: permission denied unable to create listener: Error listen tcp6 [::1]:80: bind: permission denied]
error: unable to listen on any of the requested ports: [{80 8088}]
Actually I didnā€™t add any additional config to Opta, it automatically deployed to 80 as you see
d
right, but seems that your local :80 port is busy
can you share your opta config?
a
I used these
Btw when running the first Opta run to the .env, the last stem of the runing wasnā€™t successful:
But the rest was
I got the error when running the whole thing, this screen shot shows the result when I retried it, as you see the Terraform plan only found one item pending witch is 
helm_release.metrics_server
Can this cause the issue?
@David Espejo (he/him) Any comment sir?
d
sorry, was in meetings. So, Opta should have created an Ingress resource and you should be using it to connect. Can you confirm there's an Ingress created and the status of the backing Service?
a
Hi David
thank you for your response, no worries šŸ™‚
Yes there are ingresses
I just created the project
Screen Shot 2023-03-16 at 8.58.38 PM.png
I used port 81 and it works
d
hm, so one could say that, similar to single binary deployment, 
81
in this case would be the port for non SSL connections, interesting
a
Yeah
I tested the 87 witch throws a crazy error and crashes the forward
then the 81 šŸ™‚
I finally created the project, now I need to push some workflows
d
awesome @Ahmed Laadraoui!
a
Thank you šŸ™‚
can you please tell me why the minimum task duration is around 15s even if I only return a small number in the task function?
179 Views
Open in Slack
PreviousNext
Powered by