Hello All I am trying to use flyteremote to connect to a fly

Question

Hello All I am trying to use flyteremote to connect to a flyte cluster I am able to run flytectl get projects from my terminal without any error However when I run the flyteremote as given in the example page I get SSL verify error E0123 14 24 54 984053964 2564353 Handshake failed with fatal error SSL ERROR SSL error 1000007d SSL routines OPENSSL internal CERTIFICATE VERIFY FAILED asctime 2023 01 23 14 24 54 985 name flytekit cli levelname ERROR message Non auth RPC error lt InactiveRpcError of RPC that terminated with n tstatus = StatusCode UNAVAILABLE n tdetails = failed to connect to all addresses last error UNKNOWN ipv4 10 183 180 0 443 Ssl handshake failed SSL ERROR SSL error 1000007d SSL routines OPENSSL internal CERTIFICATE VERIFY FAILED n tdebug error string = UNKNOWN failed to connect to all addresses last error UNKNOWN ipv4 10 183 180 0 443 Ssl handshake failed SSL ERROR SSL error 1000007d SSL routines OPENSSL internal CERTIFICATE VERIFY FAILED grpc status 14 created time 2023 01 23T14 24 54 98558765+02 00 n gt sleeping 200ms and retrying Kindly help < CNMKCU6FR|announcements> < CREL4QVAQ|flytekit>

Samhita Alla · Answer

Can you add `insecureSkipVerify true` to your flyte config yaml file

Sujith Samuel · Answer

I still get the same error

Sujith Samuel · Answer

admin endpoint dns lt gt insecure false insecureSkipVerify true authType ClientSecret command Set to the clientId will be used for both Pkce and ClientSecret flows Leave empty to use the value discovered through flyteAdmin s Auth discovery endpoint clientId flytepropeller Set to the location where the client secret is mounted Only needed used for `ClientSecret` flow clientSecretLocation home lt gt flyte clientsecret scopes access token

Sujith Samuel · Answer

This is how it looks like

Samhita Alla · Answer

< Yee> what could be the issue

Ketan (kumare3) · Answer

< Sujith Samuel> are you saying flytectl works

David Espejo (he/him) · Answer

Hi < Sujith Samuel> is this a sandbox environment I ve seen that same error in the past when the `KUBECONFIG`and `FLYTE CONFIG` env vars are not exported

Sujith Samuel · Answer

< Ketan kumare3 > flytectl works and < David Espejo> this is from a container environment which is connected to flyte via the flyte folder in which the clientsecret and flyte config file is present Even from a VM external to the cluster with the same flyte folder I am able to work with flytectl but when i try the remote workflow execution using python it fails asking for SSL verification

Sujith Samuel · Answer

Could you please let me know which env vars need to be exported so that I can try the same

David Espejo (he/him) · Answer

well this is for sandbox ```export KUBECONFIG=$KUBECONFIG $HOME kube config $HOME flyte k3s k3s yaml export FLYTECTL CONFIG=$HOME flyte config sandbox yaml```

Sujith Samuel · Answer

Same error still

Sujith Samuel · Answer

Can you please let me know how to specify the ca certificate location while running the flytekit remote functionality

Ketan (kumare3) · Answer

hmm let me look at the code this is odd

Ketan (kumare3) · Answer

i think you might need to pass some options <https github com flyteorg flytekit blob b6605bc95bd6f03434aeb5ca8f69eba161924ef8 flytekit clients raw py L126>

Ketan (kumare3) · Answer

or <https github com flyteorg flytekit blob b6605bc95bd6f03434aeb5ca8f69eba161924ef8 flytekit clients raw py L135>

Ketan (kumare3) · Answer

this is all grpc

Ketan (kumare3) · Answer

the thing is do not have such a setup

Ketan (kumare3) · Answer

happy to hop on a call and work with you

Sujith Samuel · Answer

Now i get this error after including the grpc creds grpc channel InactiveRpcError lt InactiveRpcError of RPC that terminated with status = StatusCode UNAUTHENTICATED details = token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken debug error string = UNKNOWN Error received from peer ipv4 10 183 180 0 443 grpc message token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken grpc status 16 created time 2023 01 23T19 43 56 291277168+02 00

Ketan (kumare3) · Answer

hmm something is odd

Sujith Samuel · Answer

I was able to cross this by enabling the redirect URL in keycloak However now the browser is opening in unix terminal and asking me to enter the user and password How can i enable this to work without this flytectl get projects does not need me to enter user id and password

David Espejo (he/him) · Answer

< Sujith Samuel> current instructions to setup Keycloak as an OiDC provider are available including configuration that has to be updated on Flyte for auth to work We can schedule a call to help you with the setup just give us a couple of options when you re available

Ketan (kumare3) · Answer

< Sujith Samuel> it should not be needed as you are using cliendId and secret

Ketan (kumare3) · Answer

sorry for the trouble lets chat

Sujith Samuel · Answer

I am fine to set up sometime with someone from your team to show this situation Some of the use cases that I am catering to require engineers to run the same workflow mutliple times with different input params

Sujith Samuel · Answer

Please help

Sujith Samuel · Answer

Can you please send me the format of hte config file to be used for flytekit Is the format different for flytekit vs flytectl

Samhita Alla · Answer

No it isn t It s the same for both of them I m guessing you ve looked at the other possible solutions suggested on Slack by the community Let s wait for Ketan to respond

Ketan (kumare3) · Answer

I think it s something to do with self signed and Grpc I think the new python Grpc channel allows something

Ketan (kumare3) · Answer

Let me take a look into golang and python and see if we can reconcile

Sujith Samuel · Answer

Trying to import flyteremote I get the below error from flytekit remote import FlyteRemote File home samuel local lib python3 8 site packages flytekit init py line 262 in lt module gt load implicit plugins File home samuel local lib python3 8 site packages flytekit init py line 256 in load implicit plugins discovered plugins = entry points group= flytekit plugins TypeError entry points got an unexpected keyword argument group

Ketan (kumare3) · Answer

what it seems you are on some older version of python

Ketan (kumare3) · Answer

is it

Sujith Samuel · Answer

which version is required for this to work

Samhita Alla · Answer

You ll need Python 3 7 at a minimum Can you also re install flytekit

Sujith Samuel · Answer

I am using 3 8

Sujith Samuel · Answer

And the below is the error that I am not getting

Sujith Samuel · Answer

asctime 2023 01 25 18 56 04 531 name flytekit remote levelname WARNING message This feature is still in beta Its interface and UX is subject to change asctime 2023 01 25 18 56 04 761 name flytekit cli levelname ERROR message Unauthenticated RPC error lt InactiveRpcError of RPC that terminated with n tstatus = StatusCode UNAUTHENTICATED n tdetails = token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken n tdebug error string = UNKNOWN Error received from peer ipv4 87 254 212 120 8080 created time 2023 01 25T18 56 04 760958564+00 00 grpc status 16 grpc message token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken n gt refreshing credentials and retrying n E0125 18 56 07 795258038 29608 Waiting for thread pool to idle before forking E0125 18 56 10 795462808 29608 Waiting for thread pool to idle before forking E0125 18 56 13 795641791 29608 Waiting for thread pool to idle before forking E0125 18 56 16 795816215 29608 Waiting for thread pool to idle before forking

Sujith Samuel · Answer

same flytectl doesnt give any issue

Sujith Samuel · Answer

I am using clientsecret

Ketan (kumare3) · Answer

< Sujith Samuel> did the entry points error go away

Ketan (kumare3) · Answer

I think definitely need to jump on a call to discuss this

Ketan (kumare3) · Answer

i am really confused

Sujith Samuel · Answer

Yes i cleaned up my VM and reinstalled from a clean ubuntu image and then now this error

Ketan (kumare3) · Answer

ohh this seems to be forking issue

Ketan (kumare3) · Answer

so you always get a failure before retrying to get creds

Ketan (kumare3) · Answer

Ketan (kumare3) · Answer

this again seems to be some python issue

Ketan (kumare3) · Answer

are you using a virtualenv

Sujith Samuel · Answer

i am running this from a k8s container

Ketan (kumare3) · Answer

Ketan (kumare3) · Answer

it seems like the version of grpc

Ketan (kumare3) · Answer

sad <https github com grpc grpc issues 31885 issuecomment 1354028785>

Ketan (kumare3) · Answer

i have seen this too

Sujith Samuel · Answer

on a direct unix box i get the below error raise NoKeyringError msg keyring errors NoKeyringError No recommended backend was available Install a recommended 3rd party backend package or install the keyrings alt package if you want to use the non recommended backends See <https pypi org project keyring> for details

Sujith Samuel · Answer

asctime 2023 01 25 21 09 23 351 name flytekit remote levelname WARNING message This feature is still in beta Its interface and UX is subject to change asctime 2023 01 25 21 09 24 059 name flytekit cli levelname ERROR message Unauthenticated RPC error lt InactiveRpcError of RPC that terminated with n tstatus = StatusCode UNAUTHENTICATED n tdetails = token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken n tdebug error string = UNKNOWN Error received from peer ipv4 10 183 180 0 443 grpc message token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken grpc status 16 created time 2023 01 25T21 09 24 058945058+02 00 n gt refreshing credentials and retrying n Traceback most recent call last File home samuel local lib python3 8 site packages flytekit clients raw py line 150 in handler return fn args kwargs File home samuel local lib python3 8 site packages flytekit clients raw py line 448 in list workflows paginated return self stub ListWorkflows resource list request metadata=self metadata File home samuel local lib python3 8 site packages grpc channel py line 946 in call return end unary response blocking state call False None File home samuel local lib python3 8 site packages grpc channel py line 849 in end unary response blocking raise InactiveRpcError state grpc channel InactiveRpcError lt InactiveRpcError of RPC that terminated with status = StatusCode UNAUTHENTICATED details = token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken debug error string = UNKNOWN Error received from peer ipv4 10 183 180 0 443 grpc message token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken grpc status 16 created time 2023 01 25T21 09 24 058945058+02 00 gt During handling of the above exception another exception occurred Traceback most recent call last File mul wf py line 15 in lt module gt workflow = remote fetch workflow name= workflow w2 workflow File home samuel local lib python3 8 site packages flytekit remote remote py line 392 in fetch workflow workflow id = get entity identifier File home samuel local lib python3 8 site packages flytekit remote remote py line 108 in get entity identifier version if version is not None else get latest version list entities method project domain name File home samuel local lib python3 8 site packages flytekit remote remote py line 84 in get latest version entity list = list entities method File home samuel local lib python3 8 site packages flytekit clients friendly py line 276 in list workflows paginated wf list = super SynchronousFlyteClient self list workflows paginated File home samuel local lib python3 8 site packages flytekit clients raw py line 159 in handler refresh handler fn args 0 File home samuel local lib python3 8 site packages flytekit clients raw py line 44 in refresh credentials standard client = credentials access get client File home samuel local lib python3 8 site packages flytekit clis auth credentials py line 20 in get client authorization client = AuthorizationClient File home samuel local lib python3 8 site packages flytekit clis auth auth py line 178 in init self refresh token = keyring get password keyring service name keyring refresh token storage key File home samuel local lib python3 8 site packages keyring core py line 55 in get password return get keyring get password service name username File home samuel local lib python3 8 site packages keyring backends fail py line 25 in get password raise NoKeyringError msg keyring errors NoKeyringError No recommended backend was available Install a recommended 3rd party backend package or install the keyrings alt package if you want to use the non recommended backends See for details samuel nmlprs dev vm ~ install flytenmlp$

Sujith Samuel · Answer

if i install the keyrings alt package it will try to open a browser on my UNIX machine and ask to enter the user and pass But I was of the opinion that clientsecret means I dont need all that

Ketan (kumare3) · Answer

Yes you are right

Ketan (kumare3) · Answer

< Eduardo Apolinario eapolinario > can you help here

Ketan (kumare3) · Answer

Also maybe we should move to the new auth flow I have built elsewhere

Sujith Samuel · Answer

Hello < Ketan kumare3 > Is there any go client to interact with flyte so that workflows can be submitted remotely If so I can use that in my own microservice

Sujith Samuel · Answer

This issue is still not resolved

Ketan (kumare3) · Answer

Ohh yes it is possible

Ketan (kumare3) · Answer

Ketan (kumare3) · Answer

But I need to figure why your python client is not working

Sujith Samuel · Answer

Any examples of the usage of go client

Sujith Samuel · Answer

We can set up a call to show you the python client

Sujith Samuel · Answer

It would be great to have an example of the usage of this flyteidl client used with the oauth flow

Ketan (kumare3) · Answer

Yea flytectl is an example

Ketan (kumare3) · Answer

And propeller

Ketan (kumare3) · Answer

It should be easy tbh

Sujith Samuel · Answer

i guessed you will say that I am looking at that code only but its kind of huge

Ketan (kumare3) · Answer

Sujith Samuel · Answer

Looks like the auth will be taken care automatically by the keyring software once I have the config file in place All i need to do is to declare the client with the correct config file and then use the admin service workflow functionalities

Sujith Samuel · Answer

Is that right

Ketan (kumare3) · Answer

Yes

Ketan (kumare3) · Answer

But this is the same with python

Sujith Samuel · Answer

Exactly but when i do the same with python Its just giving me all these error It could be issue with my environment Let me try this go thing and if that works then its just one common microservice for my users

Ketan (kumare3) · Answer

Ya but please help us fix the python client

Ketan (kumare3) · Answer

I have a python client now that is just like that golang one

Sujith Samuel · Answer

Sure We can have a call next week if that is fine with you

Ketan (kumare3) · Answer

I can open a pr with that if you want

Ketan (kumare3) · Answer

And we can try it

Ketan (kumare3) · Answer

Absolutely

Ketan (kumare3) · Answer

Anything to get this right and get you unblocked

Sujith Samuel · Answer

Sure please do the pr and I can test it in my system

Ketan (kumare3) · Answer

I will do

Sujith Samuel · Answer

I actually am having one more situation My flyte workflow execution happens on different resource systems based on the decorators that the user is specifying in his code e g if he says gpu then the execution goes to an external system In that external system what I am doing is that I am running the flyte container with pyflyte execute and its parameters as the arguments

Sujith Samuel · Answer

If I delete this workflow from the UI the workflow on cluster is deleted but since the external system gets no indication the job continues running

Ketan (kumare3) · Answer

Because you actually launch the workflow from python right

Ketan (kumare3) · Answer

This needs a backend plugin

Sujith Samuel · Answer

Yes right a backend plugin That is on our agenda but we are stretching right now We will be doing this backend plugin thing

Ketan (kumare3) · Answer

But we are making it easier to write backend plugins soon <https hackmd io pingsutw B1a Bnfqi|https hackmd io pingsutw B1a Bnfqi>

Sujith Samuel · Answer

Awesome

Ketan (kumare3) · Answer

Kevin has a prototype but he had a baby so out for a few weeks

Sujith Samuel · Answer

No worries it will take sometime for us also to move ahead on this Might be later sometime in February

Sujith Samuel · Answer

so the propeller will send message to this backend and then my backed will send signal to the external resource

Ketan (kumare3) · Answer

Ya I think flytekit can also capture the kill signal

Ketan (kumare3) · Answer

You are right the python plugin is like the backend go plugin

Sujith Samuel · Answer

Cool lets talk about this next month then I have to look at this go client now Hopefully I cna finish this one this week Lets see

Sujith Samuel · Answer

Ok Thanks a lot for your support and please keep up the good work

Ketan (kumare3) · Answer

Thank you for your support Surjith

Sujith Samuel · Answer

Would it work if i create flyteworkflows based on the user input to my microservices Once hte workflow is created then the propeller will automatically take it up and work right

Ketan (kumare3) · Answer

You have to launch an execution

Sujith Samuel · Answer

generally the controllers would pick up the CRD and then all would be taken care by the CRD processing logic I thought

Ketan (kumare3) · Answer

hmm the CRD is only created when an execution is created

Ketan (kumare3) · Answer

this way we do not pollute the cluster the workflows remain dormant in Flyteadmin and are activated and only remain in k8s for that period

Sujith Samuel · Answer

Can we please have a call this week to show this flyteremote issue This is becoming a blocker now to almost all our researchers who want to do multiple parallel workflows

Ketan (kumare3) · Answer

Absolutely

Ketan (kumare3) · Answer

When can we get on a call

Sujith Samuel · Answer

Anytime Monday or Tuesday this week night or morning Anytime is we can fix it up prior

Ketan (kumare3) · Answer

Let me dm you

Ketan (kumare3) · Answer

I also am about to create a branch with the new Grpc auth system

Sujith Samuel · Answer

I can try the branch once it s available or in beta test if you want me to

Ketan (kumare3) · Answer

It s almost there do not have self signed certs

Ketan (kumare3) · Answer

Will post something today