https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
s

Sujith Samuel

01/23/2023, 12:30 PM
Hello All, I am trying to use flyteremote to connect to a flyte cluster. I am able to run flytectl get projects from my terminal without any error. However when I run the flyteremote as given in the example page https://docs.flyte.org/projects/cookbook/en/stable/auto/remote_access/remote_workflow.html, I get SSL verify error E0123 142454.984053964 2564353 ssl_transport_security.cc:1501] Handshake failed with fatal error SSL_ERROR_SSL: error1000007dSSL routinesOPENSSL internalCERTIFICATE_VERIFY_FAILED. {"asctime": "2023-01-23 142454,985", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses; last error: UNKNOWN: ipv410.183.180.0443: Ssl handshake failed: SSL_ERROR_SSL: error1000007dSSL routinesOPENSSL internalCERTIFICATE_VERIFY_FAILED\"\n\tdebug_error_string = \"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv410.183.180.0443: Ssl handshake failed: SSL_ERROR_SSL: error1000007dSSL routinesOPENSSL internalCERTIFICATE_VERIFY_FAILED {grpc_status:14, created_time:\"2023-01-23T142454.98558765+02:00\"}\"\n>, sleeping 200ms and retrying"} Kindly help #announcements #flytekit
s

Samhita Alla

01/23/2023, 12:58 PM
Can you add 
insecureSkipVerify: true
to your flyte config.yaml file?
s

Sujith Samuel

01/23/2023, 1:15 PM
I still get the same error.
admin: endpoint: dns:///<> insecure: false insecureSkipVerify: true authType: ClientSecret #command: # Set to the clientId (will be used for both Pkce and ClientSecret flows) # Leave empty to use the value discovered through flyteAdmin's Auth discovery endpoint. clientId: flytepropeller # Set to the location where the client secret is mounted. # Only needed/used for 
ClientSecret
flow. clientSecretLocation: /home/<>/.flyte/clientsecret scopes: access_token
This is how it looks like
s

Samhita Alla

01/23/2023, 1:59 PM
@Yee, what could be the issue?
k

Ketan (kumare3)

01/23/2023, 3:18 PM
@Sujith Samuel are you saying flytectl works?
d

David Espejo (he/him)

01/23/2023, 3:37 PM
Hi @Sujith Samuel, is this a sandbox environment? I've seen that same error in the past when the `KUBECONFIG`and 
FLYTE_CONFIG
env vars are not exported
s

Sujith Samuel

01/23/2023, 4:41 PM
@Ketan (kumare3) flytectl works and @David Espejo this is from a container environment which is connected to flyte via the .flyte folder in which the clientsecret and flyte_config file is present. Even from a VM external to the cluster with the same .flyte folder, I am able to work with flytectl but when i try the remote workflow execution using python, it fails asking for SSL verification
Could you please let me know which env vars need to be exported so that I can try the same
d

David Espejo (he/him)

01/23/2023, 4:51 PM
well, this is for sandbox
Copy code
export KUBECONFIG=$KUBECONFIG:$HOME/.kube/config:$HOME/.flyte/k3s/k3s.yaml
export FLYTECTL_CONFIG=$HOME/.flyte/config-sandbox.yaml
s

Sujith Samuel

01/23/2023, 5:08 PM
Same error still
Can you please let me know how to specify the ca certificate location while running the flytekit.remote functionality
k

Ketan (kumare3)

01/23/2023, 5:13 PM
hmm let me look at the code. this is odd
this is all grpc
the thing is do not have such a setup
happy to hop on a call and work with you?
s

Sujith Samuel

01/23/2023, 5:45 PM
Now i get this error after including the grpc creds grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAUTHENTICATED details = "token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken" debug_error_string = "UNKNOWN:Error received from peer ipv410.183.180.0443 {grpc_message:"token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken", grpc_status:16, created_time:"2023-01-23T194356.291277168+02:00"}"
k

Ketan (kumare3)

01/23/2023, 5:50 PM
hmm something is odd
s

Sujith Samuel

01/23/2023, 7:25 PM
I was able to cross this by enabling the redirect URL in keycloak. However now the browser is opening in unix terminal and asking me to enter the user and password. How can i enable this to work without this. flytectl get projects does not need me to enter user id and password.
d

David Espejo (he/him)

01/23/2023, 8:11 PM
@Sujith Samuel current instructions to setup Keycloak as an OiDC provider are available here including configuration that has to be updated on Flyte for auth to work. We can schedule a call to help you with the setup, just give us a couple of options when you're available
k

Ketan (kumare3)

01/24/2023, 1:03 AM
@Sujith Samuel it should not be needed as you are using cliendId and secret
sorry for the trouble, lets chat?
s

Sujith Samuel

01/24/2023, 7:10 AM
I am fine to set up sometime with someone from your team to show this situation. Some of the use cases that I am catering to require engineers to run the same workflow mutliple times with different input params.
Please help
Can you please send me the format of hte config file to be used for flytekit. Is the format different for flytekit vs flytectl??
s

Samhita Alla

01/24/2023, 11:36 AM
No it isn't. It's the same for both of them. I'm guessing you've looked at the other possible solutions suggested on Slack by the community. Let's wait for Ketan to respond.
k

Ketan (kumare3)

01/24/2023, 2:54 PM
I think it’s something to do with self signed and Grpc. I think the new python Grpc channel allows something
Let me take a look into golang and python and see if we can reconcile
s

Sujith Samuel

01/24/2023, 8:24 PM
Trying to import flyteremote, I get the below error from flytekit.remote import FlyteRemote File "/home/samuel/.local/lib/python3.8/site-packages/flytekit/__init__.py", line 262, in <module> load_implicit_plugins() File "/home/samuel/.local/lib/python3.8/site-packages/flytekit/__init__.py", line 256, in load_implicit_plugins discovered_plugins = entry_points(group="flytekit.plugins") TypeError: entry_points() got an unexpected keyword argument 'group'
k

Ketan (kumare3)

01/24/2023, 9:13 PM
what, it seems you are on some older version of python?
is it?
s

Sujith Samuel

01/25/2023, 6:22 AM
which version is required for this to work?
s

Samhita Alla

01/25/2023, 7:49 AM
You'll need Python 3.7 at a minimum. Can you also re-install flytekit?
s

Sujith Samuel

01/25/2023, 6:56 PM
I am using 3.8
And the below is the error that I am not getting
{"asctime": "2023-01-25 185604,531", "name": "flytekit.remote", "levelname": "WARNING", "message": "This feature is still in beta. Its interface and UX is subject to change."} {"asctime": "2023-01-25 185604,761", "name": "flytekit.cli", "levelname": "ERROR", "message": "Unauthenticated RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAUTHENTICATED\n\tdetails = \"token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken\"\n\tdebug_error_string = \"UNKNOWN:Error received from peer ipv487.254.212.1208080 {created_time:\"2023-01-25T185604.760958564+00:00\", grpc_status:16, grpc_message:\"token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken\"}\"\n>, refreshing credentials and retrying\n"} E0125 185607.795258038 29608 thread_pool.cc:254] Waiting for thread pool to idle before forking E0125 185610.795462808 29608 thread_pool.cc:254] Waiting for thread pool to idle before forking E0125 185613.795641791 29608 thread_pool.cc:254] Waiting for thread pool to idle before forking E0125 185616.795816215 29608 thread_pool.cc:254] Waiting for thread pool to idle before forking
same flytectl doesnt give any issue
I am using clientsecret
k

Ketan (kumare3)

01/25/2023, 7:02 PM
@Sujith Samuel did the “entry_points” error go away?
I think definitely need to jump on a call to discuss this
i am really confused
s

Sujith Samuel

01/25/2023, 7:03 PM
Yes, i cleaned up my VM and reinstalled from a clean ubuntu image and then now this error
k

Ketan (kumare3)

01/25/2023, 7:04 PM
ohh this seems to be forking issue
so you always get a failure before retrying to get creds
this again seems to be some python issue
are you using a virtualenv
s

Sujith Samuel

01/25/2023, 7:06 PM
i am running this from a k8s container
k

Ketan (kumare3)

01/25/2023, 7:06 PM
it seems like the version of grpc
i have seen this too
s

Sujith Samuel

01/25/2023, 7:10 PM
on a direct unix box, i get the below error raise NoKeyringError(msg) keyring.errors.NoKeyringError: No recommended backend was available. Install a recommended 3rd party backend package; or, install the keyrings.alt package if you want to use the non-recommended backends. See https://pypi.org/project/keyring for details.
{"asctime": "2023-01-25 210923,351", "name": "flytekit.remote", "levelname": "WARNING", "message": "This feature is still in beta. Its interface and UX is subject to change."} {"asctime": "2023-01-25 210924,059", "name": "flytekit.cli", "levelname": "ERROR", "message": "Unauthenticated RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAUTHENTICATED\n\tdetails = \"token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken\"\n\tdebug_error_string = \"UNKNOWN:Error received from peer ipv410.183.180.0443 {grpc_message:\"token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken\", grpc_status:16, created_time:\"2023-01-25T210924.058945058+02:00\"}\"\n>, refreshing credentials and retrying\n"} Traceback (most recent call last): File "/home/samuel/.local/lib/python3.8/site-packages/flytekit/clients/raw.py", line 150, in handler return fn(*args, **kwargs) File "/home/samuel/.local/lib/python3.8/site-packages/flytekit/clients/raw.py", line 448, in list_workflows_paginated return self._stub.ListWorkflows(resource_list_request, metadata=self._metadata) File "/home/samuel/.local/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call return _end_unary_response_blocking(state, call, False, None) File "/home/samuel/.local/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAUTHENTICATED details = "token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken" debug_error_string = "UNKNOWN:Error received from peer ipv410.183.180.0443 {grpc_message:"token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken", grpc_status:16, created_time:"2023-01-25T210924.058945058+02:00"}"
During handling of the above exception, another exception occurred: Traceback (most recent call last): File "mul_wf.py", line 15, in <module> workflow = remote.fetch_workflow(name="w2.my_workflow.w2_workflow") File "/home/samuel/.local/lib/python3.8/site-packages/flytekit/remote/remote.py", line 392, in fetch_workflow workflow_id = _get_entity_identifier( File "/home/samuel/.local/lib/python3.8/site-packages/flytekit/remote/remote.py", line 108, in _get_entity_identifier version if version is not None else _get_latest_version(list_entities_method, project, domain, name), File "/home/samuel/.local/lib/python3.8/site-packages/flytekit/remote/remote.py", line 84, in _get_latest_version entity_list, _ = list_entities_method( File "/home/samuel/.local/lib/python3.8/site-packages/flytekit/clients/friendly.py", line 276, in list_workflows_paginated wf_list = super(SynchronousFlyteClient, self).list_workflows_paginated( File "/home/samuel/.local/lib/python3.8/site-packages/flytekit/clients/raw.py", line 159, in handler refresh_handler_fn(args[0]) File "/home/samuel/.local/lib/python3.8/site-packages/flytekit/clients/raw.py", line 44, in _refresh_credentials_standard client = _credentials_access.get_client( File "/home/samuel/.local/lib/python3.8/site-packages/flytekit/clis/auth/credentials.py", line 20, in get_client _authorization_client = AuthorizationClient( File "/home/samuel/.local/lib/python3.8/site-packages/flytekit/clis/auth/auth.py", line 178, in init self._refresh_token = _keyring.get_password(_keyring_service_name, _keyring_refresh_token_storage_key) File "/home/samuel/.local/lib/python3.8/site-packages/keyring/core.py", line 55, in get_password return get_keyring().get_password(service_name, username) File "/home/samuel/.local/lib/python3.8/site-packages/keyring/backends/fail.py", line 25, in get_password raise NoKeyringError(msg) keyring.errors.NoKeyringError: No recommended backend was available. Install a recommended 3rd party backend package; or, install the keyrings.alt package if you want to use the non-recommended backends. See https://pypi.org/project/keyring for details. samuel@nmlprs-dev-vm:~/install/flytenmlp$
if i install the keyrings.alt package, it will try to open a browser on my UNIX machine and ask to enter the user and pass. But I was of the opinion that clientsecret means I dont need all that
k

Ketan (kumare3)

01/25/2023, 7:16 PM
Yes you are right
@Eduardo Apolinario (eapolinario) can you help here
Also maybe we should move to the new auth flow I have built elsewhere
s

Sujith Samuel

01/28/2023, 3:58 PM
Hello @Ketan (kumare3) Is there any go client to interact with flyte so that workflows can be submitted remotely. If so I can use that in my own microservice???
This issue is still not resolved
k

Ketan (kumare3)

01/28/2023, 4:43 PM
Ohh yes it is possible
But I need to figure why your python client is not working
s

Sujith Samuel

01/28/2023, 5:04 PM
Any examples of the usage of go client???
We can set up a call to show you the python client.
It would be great to have an example of the usage of this flyteidl client used with the oauth flow.
k

Ketan (kumare3)

01/28/2023, 6:04 PM
Yea flytectl is an example
And propeller
It should be easy tbh
s

Sujith Samuel

01/28/2023, 6:04 PM
i guessed you will say that... I am looking at that code only but its kind of huge....
k

Ketan (kumare3)

01/28/2023, 6:06 PM
s

Sujith Samuel

01/28/2023, 6:17 PM
Looks like the auth will be taken care automatically by the keyring software once I have the config file in place. All i need to do is to declare the client with the correct config file and then use the admin service workflow functionalities...???
Is that right??
k

Ketan (kumare3)

01/28/2023, 6:24 PM
Yes
But this is the same with python
s

Sujith Samuel

01/28/2023, 6:28 PM
Exactly but when i do the same with python, Its just giving me all these error. It could be issue with my environment. Let me try this go thing and if that works then its just one common microservice for my users
k

Ketan (kumare3)

01/28/2023, 6:28 PM
Ya but please help us fix the python client
I have a python client now that is just like that golang one
s

Sujith Samuel

01/28/2023, 6:29 PM
Sure... We can have a call next week if that is fine with you
k

Ketan (kumare3)

01/28/2023, 6:29 PM
I can open a pr with that if you want
And we can try it
Absolutely
Anything to get this right and get you unblocked
s

Sujith Samuel

01/28/2023, 6:29 PM
Sure, please do the pr and I can test it in my system
k

Ketan (kumare3)

01/28/2023, 6:33 PM
I will do
s

Sujith Samuel

01/28/2023, 6:33 PM
I actually am having one more situation. My flyte workflow execution happens on different resource systems based on the decorators that the user is specifying in his code. e.g. if he says gpu then the execution goes to an external system. In that external system, what I am doing is that I am running the flyte container with pyflyte-execute and its parameters as the arguments
If I delete this workflow from the UI, the workflow on cluster is deleted but since the external system gets no indication, the job continues running
k

Ketan (kumare3)

01/28/2023, 6:34 PM
Because you actually launch the workflow from python right
This needs a backend plugin
s

Sujith Samuel

01/28/2023, 6:35 PM
Yes right... a backend plugin. That is on our agenda but we are stretching right now. We will be doing this backend plugin thing...
k

Ketan (kumare3)

01/28/2023, 6:36 PM
But we are making it easier to write backend plugins soon - https://hackmd.io/@pingsutw/B1a_Bnfqi
s

Sujith Samuel

01/28/2023, 6:36 PM
Awesome.
k

Ketan (kumare3)

01/28/2023, 6:36 PM
Kevin has a prototype - but he had a baby, so out for a few weeks
s

Sujith Samuel

01/28/2023, 6:38 PM
No worries, it will take sometime for us also to move ahead on this. Might be later sometime in February
so the propeller will send message to this backend and then my backed will send signal to the external resource
k

Ketan (kumare3)

01/28/2023, 6:38 PM
Ya I think flytekit can also capture the kill signal
You are right the python plugin is like the backend go plugin
s

Sujith Samuel

01/28/2023, 6:40 PM
Cool. lets talk about this next month then. I have to look at this go client now. Hopefully I cna finish this one this week. Lets see
Ok. Thanks a lot for your support and please keep up the good work.
k

Ketan (kumare3)

01/28/2023, 6:41 PM
Thank you for your support Surjith
s

Sujith Samuel

01/30/2023, 6:11 PM
Would it work if i create flyteworkflows based on the user input to my microservices. Once hte workflow is created, then the propeller will automatically take it up and work right???
k

Ketan (kumare3)

01/30/2023, 6:17 PM
You have to launch an execution
s

Sujith Samuel

01/30/2023, 6:21 PM
generally the controllers would pick up the CRD and then all would be taken care by the CRD processing logic I thought
k

Ketan (kumare3)

01/31/2023, 1:01 AM
hmm the CRD is only created when an execution is created
this way we do not pollute the cluster, the workflows remain dormant in Flyteadmin and are activated and only remain in k8s for that period
s

Sujith Samuel

02/04/2023, 6:58 PM
Can we please have a call this week to show this flyteremote issue. This is becoming a blocker now to almost all our researchers who want to do multiple parallel workflows
k

Ketan (kumare3)

02/04/2023, 7:08 PM
Absolutely
When can we get on a call
s

Sujith Samuel

02/05/2023, 3:43 PM
Anytime. Monday or Tuesday this week night or morning. Anytime is we can fix it up prior.
k

Ketan (kumare3)

02/05/2023, 4:01 PM
Let me dm you
I also am about to create a branch with the new Grpc auth system
s

Sujith Samuel

02/05/2023, 4:31 PM
I can try the branch once it’s available or in beta test if you want me to.
k

Ketan (kumare3)

02/05/2023, 4:41 PM
It’s almost there do not have self signed certs
Will post something today
11 Views
Powered by