Hii, I am currently facing an issue regarding the ...
# ask-the-community
Hii, I am currently facing an issue regarding the registration of tasks/workflows to flyteadmin, I followed @David Espejo (he/him)’s flyte-the-hard-way docs, to setup local-k3s cluster and deploy flyte-binary on it, from that I made a minor change that, instead of writing the
endpoint: "<http://minio.flyte.svc.cluster.local:9000>"
I slightly changed it to,
endpoint: "<http://minio.flyte.svc.cluster.local:30084>"
which is the nodePort of that same
service specified in this part, while accssing the node with th doing so, I've been facing error like
Copy code
[ ] Registration <http://workflows.scoop_wf.wf|workflows.scoop_wf.wf> type WORKFLOW NoneFailed with Exception Code: SYSTEM:Unknown RPC Failed, with Status: StatusCode.INTERNAL details: failed to write marshaled workflow [resource_type:WORKFLOW project:"flytesnacks" domain:"development" name:"<http://workflows.scoop_wf.wf|workflows.scoop_wf.wf>" version:"v0" ] to storage <s3://my-s3-bucket/metadata/admin/flytesnacks/development/workflows.scoop_wf.wf/v0> with err Failed to write data [9119b] to path [metadata/admin/flytesnacks/development/workflows.scoop_wf.wf/v0].: PutObject, putting object: RequestError: send request failed caused by: Put "<http://minio.flyte.svc.cluster.local:30084/my-s3-bucket/metadata/admin/flytesnacks/development/workflows.scoop_wf.wf/v0>": dial tcp i/o timeout and base container: <s3://my-s3-bucket>
while it was able to register the tasks to the minio storage, but it keeps throwing error when registering the workflow, it keeps throwing the same error when I repeated the experiment for a few times, any help would be appreciated.
@Durgai Vel, would you mind letting me know why you aren't using 9000?
Yaa sure, I have a multi node cluster environment, even if the minio pod is scheduled on anyother node, I thought it would be easier for me to just change the endpoint ip in my
file, since the minio service is
type and we can directly interact with
from that node ip and port instead of
kubectl port-forward
, also since flyte is not interacting with the service, it is interacting directly with that IP (upto my understanding), that's why! If there is any possible workaround, would be happy to change.
Flyte pods need to interact with the minio service, hence port (
) is an appropriate value to use.
Ok, Thanks @Samhita Alla, I'll try with port
, just wanted to know if there is any workaround for that. Thanks
hey @Durgai Vel. Flyte indeed interacts with the
service using K8s Service Discovery so `minio.flyte.svc.cluster.local`is an entry known inside the cluster and handled by the coreDNS instance as part of the K8s cluster. Maybe for multi-node deployment we can try using a ClusterIP instead of NodePort