https://flyte.org
Join Slack
hi <@U04H6UUE78B> I’ve followed the EKS deployment guide <https://docs.flyte.org/en/latest/deploymen...
# flyte-support
a
hi @average-finland-92144 I’ve followed the EKS deployment guide https://docs.flyte.org/en/latest/deployment/deployment/cloud_simple.html and managed to spin up a cluster (with tons of fixes) but again when I test the sample workflows with 
pyflyte run --remote hello_world.py my_wf
I get:
Copy code
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNAVAILABLE
	details: failed to connect to all addresses; last error: INTERNAL: ipv4:127.0.0.1:8088: Trying to connect an http1.x server
	Debug string UNKNOWN:failed to connect to all addresses; last error: INTERNAL: ipv4:127.0.0.1:8088: Trying to connect an http1.x server {created_time:"2023-08-17T00:00:47.690344+02:00", grpc_status:14}
Any clue?
my flyte config:
Copy code
admin:
  endpoint: dns:///localhost:8088
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 6
I forward ports with:
Copy code
kubectl -n flyte port-forward service/flyte-backend-flyte-binary-http 8088:8088
kubectl -n flyte port-forward service/flyte-backend-flyte-binary-grpc 8089:8089
no error shows up in the logs.. looks like a client side problem
a
Hi Enrico please change the port to 8089
👀 1
a
I think I did try that and got this error:
Copy code
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.INTERNAL
	details: failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials
caused by: ValidationError: Request ARN is invalid
	status code: 400, request id: 26d08b4c-11ce-4f9c-850d-ac0b9ab764fa
	Debug string UNKNOWN:Error received from peer ipv6:%5B::1%5D:8089 {created_time:"2023-08-16T23:59:27.12566+02:00", grpc_status:13, grpc_message:"failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials\ncaused by: ValidationError: Request ARN is invalid\n\tstatus code: 400, request id: 26d08b4c-11ce-4f9c-850d-ac0b9ab764fa"}
I’ll double check later
a
are you using IAM Roles for Service Accounts?
a
hi @average-finland-92144 with 
endpoint: dns:///localhost:8089
I get:
Copy code
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INTERNAL
        details = "failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials
caused by: ValidationError: Request ARN is invalid
how do I configure a role in pyflyte?
Instructions simply say list “At least one IAM role on AWS, GCP, etc. This is the role for the Flyte backend service to assume. You can provision another role for user code to assume as well” as a prerequiste. Fine I do have created a 
eksClusterRole
in IAM and that’s what EKS is using but I’m not sure how to proceed from here
a
The recommended approach is to use IRSA which requires a trust relationship and association between the Service Account, IAM Role, Policy and the EKS cluster itself. I hope these instructions are helpful for you: https://github.com/davidmirror-ops/flyte-the-hard-way/blob/main/docs/03-roles-service-accounts.md
👀 1
a
hi @average-finland-92144 yes that link should be useful! gratitude thank you I’ll restart my deployment again from scratch because those instructions look different than the official ones: https://docs.flyte.org/en/latest/deployment/deployment/cloud_simple.html Let me just triple-check with you before I set out on a new endeavour. Are the instructions at https://github.com/davidmirror-ops/flyte-the-hard-way/ the best way to deploy Flyte on K8s? Meaning, most comprehensive and tested? The reason I’m asking is I’ve been going in circles for a while trying to follow the official docs lolcry 🫠 🐬 After I get this deployed. How to deploy my own custom Agent (see code here)? In previous attempts I’ve followed these official instructions but at this point I’m not 100% sure that’ll work with the 
flyte-the-hard-way
approach. Can you point me to the best way on how to deploy my custom agents? - @steep-jackal-21573 since you’ve authored the docs page on this topic, can you share your thoughts? Thank you so much folks!
@tall-lock-23197 I’ve got stuck a number of times trying to follow the official docs (see message 👆). Do you have plans to facelift the docs somehow? Can you or anybody on your team help me closely in deploying Flyte on K8s?! Who’s the best person to talk to? Thanks!
a
Can you or anybody on your team help me closely in deploying Flyte on K8s?
I can help you, and sorry for the inconveniences so far. 🙂
t
@adventurous-van-8963, I'm sorry if the documentation is unclear. If you find any mistakes, could you please create PRs to update the docs? Thanks!
@glamorous-carpet-83516, could you assist Enrico with deploying a custom agent on his Flyte deployment, please?
And @average-finland-92144, could we include a link to "flyte the hard way" somewhere in the deployment guide? WDYT?
s
I don't exactly know how different docs are organized as I merely contributed to the agent page going together with a PR change, and https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/extend_flyte/agent_service.html#update-flyteagent looks correct to me.
👍 1
a
And @average-finland-92144, could we include a link to "flyte the hard way" somewhere in the deployment guide? WDYT?
agree, will work on this
❤️ 2
43 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by