https://flyte.org logo
Join Slack
Powered by
Hi Flyte Community, Greetings of the Day, I have a...
# flyte-support
l
Hi Flyte Community, Greetings of the Day, I have a query related to flyte. I am using flyte helm chart for my Single Node cluster. with minio and postgres dependencies already installed on cluster in flyte namespace. Also ingress is in in ingress-nginx namespace. I am also using external Auth Server and OIDC with Azure AD. Currently Two problems are there in my flyte deployment. Problem 1: My flytescheduler is giving me such error:
Copy code
panic: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"

goroutine 1 [running]:
Note: I already try to solve by adding few annotations in my ingress Those annotations are: 
enabled: true
webpackHMR: false
separateGrpcIngress: true
separateGrpcIngressAnnotations:
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
annotations:
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
<http://ingress.kubernetes.io/rewrite-target|ingress.kubernetes.io/rewrite-target>: /
<http://nginx.ingress.kubernetes.io/proxy-buffer-size|nginx.ingress.kubernetes.io/proxy-buffer-size>: "128k"
<http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "600"
<http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "600"
<http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
<http://nginx.ingress.kubernetes.io/affinity|nginx.ingress.kubernetes.io/affinity>: "cookie"
<http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
Problem 2: In my system port 9000 is busy with some other application running on it. So I am using 9200 and 9201 as a minio service local-port. Now I already setup minio endpoint in storage section with 9200 port and also In k8s plugin's default env section. But Still I recieve this error during pyflyte run command in which minio endpoint is showing 9000 as port number. Now how can I change that?
Copy code
warnings.warn(
Failed with Exception Code: USER:ValueError
USER:ValueError: error=Value error!  Received: 400. Request to send data <http://minio.flyte.svc.cluster.local:9000/flyte-bucket/flytesnacks/development/555TNYN3Q6EC7NMQTA3USVCQSA%3D%3D%3D%3D%3D%3D/script_mode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20240422%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240422T074930Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost%3Bx-amz-meta-flytecontentmd5&X-Amz-Signature=e517e6a6972f014e24769616c3eb176e78456faaaf8df5177a46f36ab3919bff> failed.
Response: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>MissingFields</Code><Message>Missing fields in request.</Message><Key>flytesnacks/development/555TNYN3Q6EC7NMQTA3USVCQSA======/script_mode.tar.gz</Key><BucketName>flyte-bucket</BucketName><Resource>/flyte-bucket/flytesnacks/development/555TNYN3Q6EC7NMQTA3USVCQSA======/script_mode.tar.gz</Resource><Region>gd-eu-muc9</Region><RequestId>17C88AB570F9715A</RequestId><HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId></Error>
t
fyi and unrelated, but if you’re using just a single node cluster there’s no reason not to use the flyte-binary helm chart. that runs flyte as just one component in k8s rather than the six or so that the flyte-core helm chart runs.
wrt to the minio issue, it thinks something is still running on 9000. I would go through the generated sample manifest https://github.com/flyteorg/flyte/blob/master/deployment/sandbox/flyte_helm_generated.yaml
and just make sure that all references to minio/9000 are set to the ports you want
i don’t think i’ve seen frame too large before, but unf I suspect it’s a networking layer issue.
a
@loud-keyboard-72751 also, please check your local 
config.yaml
file to see there is not storage config pointing to the `9000`port
l
I checked it @average-finland-92144...Storage section there does not point to 9000 port
Hi @thankful-minister-83577 As you told me to look the yaml file after helm install. I found out there is some wrong configuration it took for the storage section. I found this in the storage.yaml file.
But I don't know how to change the configuration inside stow via helm values file. I tried every way but these config didn't change. Can you suggest some proper yaml format for storage section to connect to minio
I also tried @average-finland-92144 the hard way flyte setup documentation. It either not worked as in that It has it's default sandbox configuration. Also I try to change local config.yaml file by adding storage section It is also not worked.
t
are you still using the flyte/flyte-core charts?
l
Yes @thankful-minister-83577...we actually move to multi node cluster when we get familiar with flyte. so it will be good for us to start with flyte-core
a
@loud-keyboard-72751 the key 
type: sandbox
is converted to 
minio
by the helpers (source) so you can use it. There you can see also the problem: the 
9000
port is hardcoded here What if you try 
type:custom
?
l
I tried custom but didn't know what to write inside custom:{}...Like what's the proper way to write accessKey secretKey and authType and endpoint there....If you know I can try that
a
according to the templates, it will just render whatever you put there, so I'd keep the same format/indentation, just setting 
type: custom
l
Can you provide me a sample It will be more clear
a
Copy code
storage:
  type: custom
  container: "flyte-bucket"
  stow:
    kind: s3
    config:
       access_key_id: xyz
...
I haven't personally tried this, and ideally an Issue must be created to track this problem of hardcoded port
l
I will create a PR for that and also I will try this first
t
the ports and such were set mostly to reduce the possibility of issues in the default cases.
of course feel free to copy paste and configure as needed for special deployments.
l
Thanks @thankful-minister-83577 and @average-finland-92144 Using custom type my problem 2 is solved. Thanks a lot of the help. Now I am one step ahead in my problems.
But I think Problem 1 has effects on flytepropeller as well. It didn't start pods after running pyflyte.
t
yeah could you start another thread for the frame size issue? maybe someone has an idea.
i don’t think that is a flyte specific thing.
could you also include any details you have about your networking layer
this looks like eventing that propeller is sending to admin right?
l
Yes Sure...It is something related to GRPC . i saw some other applications who are using GRPC backend has the same issues posted on their channels
Yes this is propeller sending to admin I think...Otherwise that particular build workflow did not processed to minio and didn't show in flyteconsole.
Hi.......First Thanks to everyone.. and special thatnks to you guys.@thankful-minister-83577 and @average-finland-92144. flyte-core is successfully working in my work Infrastructure. All the pods and authentication is correctly working, Besides that pyflyte and flyteconsole and flytectl all three are working fine as well.
Just for the curiosity I try to deploy flyte-binary as well and, I bumped into this error. Which I never see before when i am trying to deploy flyte binary using helm. Please have a look at that
a
@loud-keyboard-72751 but are you trying to deploy flyte-binary to a different K8s cluster and different Ingress?
l
Hi @average-finland-92144 It's working now. My Authentication towards flytepropeller side wasn't completed successfully that's the cause of the error. And I found out flytepropeller uses clientCredential auth Type so requires specific permission towards AzureAD. And when I do that and also set httpProxyURL and scopes parameter in the values file under inline section like this. Everything works and Even flyte-core is working as well now.
Copy code
inline:
  admin:
   httpProxyURL: "<PROXY_VALUE>"
   scopes:
     - api://<clientID>/.default
a
Thanks for sharing @loud-keyboard-72751. Any chance you could maybe share your learnings during the process? I think using EntraID as external auth server with a proxy in the middle is a use case that we haven't covered. Even a simple hackmd can work and we can get it added to the docs
l
I maintain a proper doc on setting up flyte binary and flyte with azure EntraID for my own learning I will share that with you
gratitude thank you 1
🙌🏽 1
🙇🏽 1
Also can you help me with How can I utilise GPU through flyte?
a
sure, let's start a new, public thread
l
Sure
104 Views
Open in Slack
PreviousNext
Powered by