Hi Flyte community, When setting up the flyte-binary helm resources and attempting to run commands (register/run) through jflyte or pyflyte, I see the following error:

Max retries exceeded with url: /my-s3-bucket/flytesnacks/development... 'Connection to 10.110.44.3 timed out

I'm able to create a new project through flytectl and see it in the flyte console. I've installed the flyte-binary helm chart with the following overrides:

configuration:
  database:
    port: 5432
    dbname: flyteadmin
    host: 10.99.211.252
  storage:
    metadataContainer: my-s3-bucket
    userDataContainer: my-s3-bucket
    provider: s3
    providerConfig:
      s3:
        disableSSL: true
        v2Signing: false
        endpoint: <http://10.110.44.3:9000>
        authType: accesskey
        accessKey: minio
        secretKey: miniostorage
  plugins:
  # All k8s plugins default configuration
    k8s:
      inject-finalizer: true
      default-env-vars:
        - AWS_METADATA_SERVICE_TIMEOUT: 5
        - AWS_METADATA_SERVICE_NUM_ATTEMPTS: 20
        - FLYTE_AWS_ENDPOINT: "<http://10.110.44.3:9000>"
        - FLYTE_AWS_ACCESS_KEY_ID: minio
        - FLYTE_AWS_SECRET_ACCESS_KEY: miniostorage

I'm able to connect the minio client CLI to localhost:9000 after portforwarding the minio service (from 9000->9000). I'm also able to successfully curl

<http://10.110.44.3:9000>

from the flyte-binary pod using

kubectl exec

. Would folks happen to know what might be happening here or what I could do to move forward in this investigation?

could you share your flytectl config

This is my flytectl config

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///localhost:8089
  insecure: true
logger:
  show-source: true
  level: 0

you shouldn’t need to port-forward the minio

This is the full log:

./bin/pyflyte --config ~/.flyte/config.yaml run --remote core/flyte_basics/hello_world.py my_wf
Traceback (most recent call last):
  File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/opt/homebrew/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/homebrew/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/homebrew/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/homebrew/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/opt/homebrew/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connection.py", line 179, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x1568dc940>, 'Connection to 10.110.44.3 timed out. (connect timeout=None)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.110.44.3', port=9000): Max retries exceeded with url: /my-s3-bucket/flytesnacks/development/YYHE6OSLHL2BT5PSBO3SQXDVSQ%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230218%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230218T004808Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=77ab13284c697d4a316b991677384f2b93ae827edf6edafbcf7e20be02f320a8 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x1568dc940>, 'Connection to 10.110.44.3 timed out. (connect timeout=None)'))

what’s your signedUrl config

Thank you for the suggestions, Kevin! Unfortunately, still seeing the same error.. I will try to instead integrate Flyte with a cloud provider storage solution