Hii, I am currently facing an issue regarding the registration of tasks/workflows to flyteadmin, I followed @David Espejo (he/him)’s flyte-the-hard-way docs, to setup local-k3s cluster and deploy flyte-binary on it, from that I made a minor change that, instead of writing the

endpoint: "<http://minio.flyte.svc.cluster.local:9000>"

I slightly changed it to,

endpoint: "<http://minio.flyte.svc.cluster.local:30084>"

which is the nodePort of that same

minio

service specified in this part, while accssing the node with th doing so, I've been facing error like

[ ] Registration <http://workflows.scoop_wf.wf|workflows.scoop_wf.wf> type WORKFLOW NoneFailed with Exception Code: SYSTEM:Unknown RPC Failed, with Status: StatusCode.INTERNAL details: failed to write marshaled workflow [resource_type:WORKFLOW project:"flytesnacks" domain:"development" name:"<http://workflows.scoop_wf.wf|workflows.scoop_wf.wf>" version:"v0" ] to storage <s3://my-s3-bucket/metadata/admin/flytesnacks/development/workflows.scoop_wf.wf/v0> with err Failed to write data [9119b] to path [metadata/admin/flytesnacks/development/workflows.scoop_wf.wf/v0].: PutObject, putting object: RequestError: send request failed caused by: Put "<http://minio.flyte.svc.cluster.local:30084/my-s3-bucket/metadata/admin/flytesnacks/development/workflows.scoop_wf.wf/v0>": dial tcp 10.43.158.135:30084: i/o timeout and base container: <s3://my-s3-bucket>

while it was able to register the tasks to the minio storage, but it keeps throwing error when registering the workflow, it keeps throwing the same error when I repeated the experiment for a few times, any help would be appreciated.

@Durgai Vel, would you mind letting me know why you aren't using 9000?

Yaa sure, I have a multi node cluster environment, even if the minio pod is scheduled on anyother node, I thought it would be easier for me to just change the endpoint ip in my

/etc/hosts

file, since the minio service is

NodePort

type and we can directly interact with

minio

from that node ip and port instead of

kubectl port-forward

, also since flyte is not interacting with the service, it is interacting directly with that IP (upto my understanding), that's why! If there is any possible workaround, would be happy to change.

Flyte pods need to interact with the minio service, hence port (

9000

) is an appropriate value to use.

Ok, Thanks @Samhita Alla, I'll try with port

9000

, just wanted to know if there is any workaround for that. Thanks

hey @Durgai Vel. Flyte indeed interacts with the

minio

service using K8s Service Discovery so `minio.flyte.svc.cluster.local`is an entry known inside the cluster and handled by the coreDNS instance as part of the K8s cluster. Maybe for multi-node deployment we can try using a ClusterIP instead of NodePort