https://flyte.org logo
#ask-the-community
Title
# ask-the-community
e

Enrico Rotundo

08/16/2023, 10:02 PM
hi @David Espejo (he/him) I’ve followed the EKS deployment guide https://docs.flyte.org/en/latest/deployment/deployment/cloud_simple.html and managed to spin up a cluster (with tons of fixes) but again when I test the sample workflows with
pyflyte run --remote hello_world.py my_wf
I get:
Copy code
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNAVAILABLE
	details: failed to connect to all addresses; last error: INTERNAL: ipv4:127.0.0.1:8088: Trying to connect an http1.x server
	Debug string UNKNOWN:failed to connect to all addresses; last error: INTERNAL: ipv4:127.0.0.1:8088: Trying to connect an http1.x server {created_time:"2023-08-17T00:00:47.690344+02:00", grpc_status:14}
Any clue?
my flyte config:
Copy code
admin:
  endpoint: dns:///localhost:8088
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 6
I forward ports with:
Copy code
kubectl -n flyte port-forward service/flyte-backend-flyte-binary-http 8088:8088
kubectl -n flyte port-forward service/flyte-backend-flyte-binary-grpc 8089:8089
no error shows up in the logs.. looks like a client side problem
d

David Espejo (he/him)

08/16/2023, 10:44 PM
Hi Enrico please change the port to 8089
e

Enrico Rotundo

08/17/2023, 7:35 AM
I think I did try that and got this error:
Copy code
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.INTERNAL
	details: failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials
caused by: ValidationError: Request ARN is invalid
	status code: 400, request id: 26d08b4c-11ce-4f9c-850d-ac0b9ab764fa
	Debug string UNKNOWN:Error received from peer ipv6:%5B::1%5D:8089 {created_time:"2023-08-16T23:59:27.12566+02:00", grpc_status:13, grpc_message:"failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials\ncaused by: ValidationError: Request ARN is invalid\n\tstatus code: 400, request id: 26d08b4c-11ce-4f9c-850d-ac0b9ab764fa"}
I’ll double check later
d

David Espejo (he/him)

08/17/2023, 4:07 PM
are you using IAM Roles for Service Accounts?
e

Enrico Rotundo

08/18/2023, 2:03 PM
hi @David Espejo (he/him) with
endpoint: dns:///localhost:8089
I get:
Copy code
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INTERNAL
        details = "failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials
caused by: ValidationError: Request ARN is invalid
how do I configure a role in pyflyte?
Instructions simply say list “At least one IAM role on AWS, GCP, etc. This is the role for the Flyte backend service to assume. You can provision another role for user code to assume as well” as a prerequiste. Fine I do have created a
eksClusterRole
in IAM and that’s what EKS is using but I’m not sure how to proceed from here
d

David Espejo (he/him)

08/18/2023, 2:10 PM
The recommended approach is to use IRSA which requires a trust relationship and association between the Service Account, IAM Role, Policy and the EKS cluster itself. I hope these instructions are helpful for you: https://github.com/davidmirror-ops/flyte-the-hard-way/blob/main/docs/03-roles-service-accounts.md
e

Enrico Rotundo

08/18/2023, 2:48 PM
hi @David Espejo (he/him) yes that link should be useful! gratitude thank you I’ll restart my deployment again from scratch because those instructions look different than the official ones: https://docs.flyte.org/en/latest/deployment/deployment/cloud_simple.html Let me just triple-check with you before I set out on a new endeavour. Are the instructions at https://github.com/davidmirror-ops/flyte-the-hard-way/ the best way to deploy Flyte on K8s? Meaning, most comprehensive and tested? The reason I’m asking is I’ve been going in circles for a while trying to follow the official docs lolcry 🫠 🐬 After I get this deployed. How to deploy my own custom Agent (see code here)? In previous attempts I’ve followed these official instructions but at this point I’m not 100% sure that’ll work with the
flyte-the-hard-way
approach. Can you point me to the best way on how to deploy my custom agents? - @honnix since you’ve authored the docs page on this topic, can you share your thoughts? Thank you so much folks!
@Samhita Alla I’ve got stuck a number of times trying to follow the official docs (see message 👆). Do you have plans to facelift the docs somehow? Can you or anybody on your team help me closely in deploying Flyte on K8s?! Who’s the best person to talk to? Thanks!
d

David Espejo (he/him)

08/18/2023, 3:11 PM
Can you or anybody on your team help me closely in deploying Flyte on K8s?
I can help you, and sorry for the inconveniences so far. 🙂
s

Samhita Alla

08/19/2023, 9:52 AM
@Enrico Rotundo, I'm sorry if the documentation is unclear. If you find any mistakes, could you please create PRs to update the docs? Thanks!
@Kevin Su, could you assist Enrico with deploying a custom agent on his Flyte deployment, please?
And @David Espejo (he/him), could we include a link to "flyte the hard way" somewhere in the deployment guide? WDYT?
h

honnix

08/21/2023, 6:58 AM
I don't exactly know how different docs are organized as I merely contributed to the agent page going together with a PR change, and https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/extend_flyte/agent_service.html#update-flyteagent looks correct to me.
d

David Espejo (he/him)

08/22/2023, 5:28 PM
And @David Espejo (he/him), could we include a link to "flyte the hard way" somewhere in the deployment guide? WDYT?
agree, will work on this
17 Views