Somewhat new to the ins and outs of implementing Flyte and h

Question

Somewhat new to the ins and outs of implementing Flyte and have been working through the Flyte the hard way binary setup I ve gotten to the point of attempting the hello world workflow but am getting the below errors when I attempt to execute it as per the instructions ```~ flyte flytesnacks examples basics basics master ᐅ pyflyte run remote hello world py my wf E0724 11 27 28 869290000 4558732800 Error parsing content type metadata error=invalid value key=content type E0724 11 27 28 891411000 4558732800 Error parsing content type metadata error=invalid value key=content type E0724 11 27 28 914052000 4558732800 Error parsing content type metadata error=invalid value key=content type E0724 11 27 28 938491000 4558732800 Error parsing content type metadata error=invalid value key=content type Failed with Exception Code SYSTEM Unknown RPC Failed with Status StatusCode UNKNOWN details Stream removed Debug string UNKNOWN Error received from peer grpc message Stream removed grpc status 2 created time 2023 07 24T11 27 28 938614 04 00 ``` Didn t see anything upthread and was wondering if anyone has run into this before The pod and ingress seem good And here s my basic config yaml ```admin For GRPC endpoints you might want to use dns flyte myexample com endpoint dns lt my flyte backend flyte binary grpc address gt Replace with your domain name authType Pkce insecure false insecureSkipVerify true logger show source true level 6```

Yee · Answer

can you pip show grpcio

Matt Coan · Answer

Sure thing ```Name grpcio Version 1 56 0 Summary HTTP 2 based RPC framework Home page Author The gRPC Authors Author email License Apache License 2 0 Location usr local lib python3 10 site packages Requires Required by flytekit grpcio status```

Yee · Answer

can you try downgrading to 1 54 x

Yee · Answer

still trying to fully understand this particular hack parser issue but we ve been seeing it come up randomly

Matt Coan · Answer

Sure lemme give that a shot

Matt Coan · Answer

Downgraded to ```Name grpcio Version 1 54 0 Summary HTTP 2 based RPC framework Home page Author The gRPC Authors Author email License Apache License 2 0 Location usr local lib python3 10 site packages Requires Required by flytekit grpcio status```

Matt Coan · Answer

Still get the error but instead of ```stream removed``` It s now a 404 ```~ flyte flytesnacks examples basics basics master ᐅ pyflyte run remote hello world py my wf E0724 12 47 06 765915000 4445138432 Error parsing metadata error=invalid value key=content type value=text plain charset=utf 8 E0724 12 47 06 792559000 4445138432 Error parsing metadata error=invalid value key=content type value=text plain charset=utf 8 E0724 12 47 06 815735000 4445138432 Error parsing metadata error=invalid value key=content type value=text plain charset=utf 8 E0724 12 47 06 840782000 4445138432 Error parsing metadata error=invalid value key=content type value=text plain charset=utf 8 Failed with Exception Code SYSTEM Unknown RPC Failed with Status StatusCode UNIMPLEMENTED details Received http2 header with status 404 Debug string UNKNOWN Error received from peer grpc message Received http2 header with status 404 grpc status 12 created time 2023 07 24T12 47 06 840918 04 00 ```

Yee · Answer

what backend version are you running

Yee · Answer

unimplemented is usually a pretty simple thing to fix

Yee · Answer

usually that means the back end needs to be upgraded

Matt Coan · Answer

Ah that would be great if I was just installing something stale and could upgrade ```NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION flyte backend flyte 2 2023 07 21 15 45 37 537995 0400 EDT deployed flyte binary v1 7 0 1 16 0```

Matt Coan · Answer

In my case I just took what the ```helm install flyte backend flyteorg flyte binary namespace flyte values eks starter yaml create namespace``` command gave me

Matt Coan · Answer

Then ```helm upgrade flyte backend flyteorg flyte binary n flyte values eks production yaml install```

Yee · Answer

are there logs on the flyte pod side that look relevant

Matt Coan · Answer

I don t see anything that jumps out

Matt Coan · Answer

Sorry had to jump into meetings for a bit

Yee · Answer

we re going to look into the parser issue later but it still makes me uneasy to see that there

Yee · Answer

could you try a couple other versions of grpcio <https pypi org project grpcio history>

Yee · Answer

see if it persists

Yee · Answer

the error logs not the unimplemented bit

Matt Coan · Answer

Sure thing

Yee · Answer

for unimplemented i wonder what it s hitting

Yee · Answer

run your command with `GRPC VERBOSITY=debug GRPC TRACE=all` then | grep tcp posix

Matt Coan · Answer

Yeah just found set those

Matt Coan · Answer

So on grpcio 1 53 0 with the debug on get a lot more detail however not sure what to make of it

Yee · Answer

this is with the grep

Matt Coan · Answer

Ah no I didn t see it running it with it now

Yee · Answer

can you confirm with a `flytectl get projects` command that you have access

Yee · Answer

if you can list projects then somehow your data proxy service isn t being installed

Matt Coan · Answer

Yeah that s actually throwing errors too flytectl get projects ``` json src viper go 398 level debug msg Config section storage updated No update handler registered ts 2023 07 24T15 53 54 04 00 json src viper go 398 level debug msg Config section root updated No update handler registered ts 2023 07 24T15 53 54 04 00 json src viper go 400 level debug msg Config section admin updated Firing updated event ts 2023 07 24T15 53 54 04 00 json src viper go 398 level debug msg Config section files updated No update handler registered ts 2023 07 24T15 53 54 04 00 json src viper go 398 level debug msg Config section console updated No update handler registered ts 2023 07 24T15 53 54 04 00 json src client go 145 level warning msg using insecureSkipVerify Server s certificate chain and host name wont be verified Caution shouldn t be used for production usecases ts 2023 07 24T15 53 54 04 00 json src client go 63 level info msg Initialized Admin client ts 2023 07 24T15 53 54 04 00 json src auth interceptor go 86 level debug msg Request failed due to rpc error code = Unimplemented desc = unexpected HTTP status code received from server 404 Not Found transport received unexpected content type text plain charset=utf 8 If it s an unauthenticated error we will attempt to establish an authenticated context ts 2023 07 24T15 53 54 04 00 Error rpc error code = Unimplemented desc = unexpected HTTP status code received from server 404 Not Found transport received unexpected content type text plain charset=utf 8 json src main go 13 level error msg rpc error code = Unimplemented desc = unexpected HTTP status code received from server 404 Not Found transport received unexpected content type text plain charset=utf 8 ts 2023 07 24T15 53 54 04 00 ```

Yee · Answer

can you port forward the grpc port from admin to localhost and then try

Matt Coan · Answer

So it does allow me to forward the port and sits there and listens I point my flyte config to it ```admin For GRPC endpoints you might want to use dns flyte myexample com endpoint dns 127 0 0 1 8089 Replace with your domain name authType Pkce insecure true insecureSkipVerify true logger show source true level 10``` And if I try to execute the hello world workflow I get ```Failed with Exception Code SYSTEM Unknown RPC Failed with Status StatusCode INTERNAL details failed to create a signed url Error WebIdentityErr failed to retrieve credentials caused by AccessDenied Not authorized to perform sts AssumeRoleWithWebIdentity status code 403 request id 7b26a76e 215c 4492 80d7 62e458acc320 Debug string UNKNOWN Error received from peer ipv4 127 0 0 1 8089 grpc message failed to create a signed url Error WebIdentityErr failed to retrieve credentials ncaused by AccessDenied Not authorized to perform sts AssumeRoleWithWebIdentity n tstatus code 403 request id 7b26a76e 215c 4492 80d7 62e458acc320 grpc status 13 created time 2023 07 24T16 06 24 281082 04 00 ```

Matt Coan · Answer

But interestingly flytectl get projects works

Matt Coan · Answer

And it for sure handles the connection attempts locally

Yee · Answer

can you grant the admin role more perms

Matt Coan · Answer

Yeah I did try that originally giving it full blown admin access temporarily to no avail I ve made a bit of progress this morning though I stopped using the original cert and generated a new one and uploaded it to ACM I grabbed the cert out and updated my flyte config to point to it ```caCertFilePath Users mattcoan flyte crt out``` If I run the hello world workflow with this cert in the mix config I get ```SSLError SSLCertVerificationError 1 SSL CERTIFICATE VERIFY FAILED certificate verify failed unable to get local issuer certificate ssl c 997 ``` However I m now able to skip SSL verification it which I know isn t great in my config ```insecureSkipVerify true``` And with that set can get the hello world workflow to run ``` usr local lib python3 10 site packages urllib3 connectionpool py 1045 InsecureRequestWarning Unverified HTTPS request is being made to host Adding certificate verification is strongly advised See warnings warn Go to to see execution in the console ``` So at least I know that once I m beyond auth the mechanics behind everything else work Just a matter of figuring out why it can t find this cert locally I think If you have any pointers in that direction I d love to hear them