Jimmy Du
02/18/2023, 12:57 AMMax retries exceeded with url: /my-s3-bucket/flytesnacks/development... 'Connection to 10.110.44.3 timed out
I'm able to create a new project through flytectl and see it in the flyte console.
I've installed the flyte-binary helm chart with the following overrides:
configuration:
database:
port: 5432
dbname: flyteadmin
host: 10.99.211.252
storage:
metadataContainer: my-s3-bucket
userDataContainer: my-s3-bucket
provider: s3
providerConfig:
s3:
disableSSL: true
v2Signing: false
endpoint: <http://10.110.44.3:9000>
authType: accesskey
accessKey: minio
secretKey: miniostorage
plugins:
# All k8s plugins default configuration
k8s:
inject-finalizer: true
default-env-vars:
- AWS_METADATA_SERVICE_TIMEOUT: 5
- AWS_METADATA_SERVICE_NUM_ATTEMPTS: 20
- FLYTE_AWS_ENDPOINT: "<http://10.110.44.3:9000>"
- FLYTE_AWS_ACCESS_KEY_ID: minio
- FLYTE_AWS_SECRET_ACCESS_KEY: miniostorage
I'm able to connect the minio client CLI to localhost:9000 after portforwarding the minio service (from 9000->9000). I'm also able to successfully curl <http://10.110.44.3:9000>
from the flyte-binary pod using kubectl exec
.
Would folks happen to know what might be happening here or what I could do to move forward in this investigation?Kevin Su
02/18/2023, 1:00 AMJimmy Du
02/18/2023, 1:01 AMadmin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///localhost:8089
insecure: true
logger:
show-source: true
level: 0
Kevin Su
02/18/2023, 1:03 AMJimmy Du
02/18/2023, 1:05 AM./bin/pyflyte --config ~/.flyte/config.yaml run --remote core/flyte_basics/hello_world.py my_wf
Traceback (most recent call last):
File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/opt/homebrew/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/homebrew/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/homebrew/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/homebrew/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1037, in _send_output
self.send(msg)
File "/opt/homebrew/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 975, in send
self.connect()
File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connection.py", line 179, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x1568dc940>, 'Connection to 10.110.44.3 timed out. (connect timeout=None)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/Users/jimmydu/git_repo/flytesnacks/cookbook/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.110.44.3', port=9000): Max retries exceeded with url: /my-s3-bucket/flytesnacks/development/YYHE6OSLHL2BT5PSBO3SQXDVSQ%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230218%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230218T004808Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=77ab13284c697d4a316b991677384f2b93ae827edf6edafbcf7e20be02f320a8 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x1568dc940>, 'Connection to 10.110.44.3 timed out. (connect timeout=None)'))
Kevin Su
02/18/2023, 4:15 AMsignedUrl:
stowConfigOverride:
endpoint: <http://10.110.44.3:9000>
Jimmy Du
02/23/2023, 11:08 PM