late-garage-39497
05/18/2022, 5:31 PMflytectl sandbox start
and exported the env vars for kube and flyte configuration.
I'm trying to run the hello world example from flytesnacks and I've created a virtual env using poetry, in which I installed pyflyte (1.0.2). I can run it locally no problem, but when I add the remote
flag, I get a connection reset by peer error.
Any ideas what is wrong on my setup?
I'm on WSL2thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
ls ~/.flyte
and if there’s a config.yaml in there can you cat it?thankful-minister-83577
thankful-minister-83577
freezing-airport-6809
late-garage-39497
05/19/2022, 7:55 AMconfig-sandbox.yaml
which was created by the flytectl command
late-garage-39497
05/19/2022, 7:55 AMTraceback (most recent call last):
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 443, in put_data
DataPersistencePlugins.find_plugin(remote_path)().put(local_path, remote_path, recursive=is_multipart)
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/extras/persistence/http.py", line 67, in put
rsp = requests.put(
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/api.py", line 132, in put
return request('put', url, data=data, **kwargs)
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/bin/pyflyte", line 8, in <module>
sys.exit(main())
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/clis/sdk_in_container/run.py", line 473, in _run
wf = remote.register_script(
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/remote/remote.py", line 536, in register_script
upload_location, md5_bytes = fast_register_single_script(
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/tools/script_mode.py", line 118, in fast_register_single_script
flyte_ctx.file_access.put_data(archive_fname, upload_location.signed_url)
File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 445, in put_data
raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmpx5p7zdl2/script_mode.tar.gz to <http://localhost:30084/my-s3-bucket/gp/flytesnacks/development/ZVTXNVVEVCVG6EWW4ECBUKYBCU%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20220518%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220518T172729Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=458ac5b8322ade953b25890b321141ae04956eda640d11674cde26dc743b906a> (recursive=False).
Full Error hereglamorous-carpet-83516
05/19/2022, 7:58 AMlate-garage-39497
05/19/2022, 7:58 AMadmin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: localhost:30081
#authType: Pkce
insecure: true
logger:
show-source: true
level: 0
I know see that I removed the authenticationglamorous-carpet-83516
05/19/2022, 8:05 AMnetstat -anvp tcp | awk 'NR<3 || /LISTEN/
? port 30081, 30082, 30084 should be open.late-garage-39497
05/19/2022, 8:10 AMProto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:30080 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:30081 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:30082 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:30084 0.0.0.0:* LISTEN
they seem to belate-garage-39497
05/19/2022, 8:44 AMlate-garage-39497
05/19/2022, 8:54 AMContainers:
minio:
Container ID: <docker://b97239c7282e279b27ecdd7b59ccde6dfa9224cdee6581ba7d6d5362fc335ad>c
Image: <http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>
Image ID: <docker-pullable://ecr.flyte.org/bitnami/minio@sha256:547a4d0fdc82d5213fef3f4f7215fee788398238c5184a8555837bd3f649525e>
Ports: 9000/TCP, 9001/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 19 May 2022 09:49:18 +0100
Finished: Thu, 19 May 2022 09:49:28 +0100
Ready: False
Restart Count: 13
Limits:
cpu: 200m
memory: 512Mi
Requests:
cpu: 10m
memory: 128Mi
Environment:
MINIO_ACCESS_KEY: minio
MINIO_SECRET_KEY: miniostorage
MINIO_DEFAULT_BUCKETS: my-s3-bucket
Mounts:
/data from minio-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zbvpf (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
minio-storage:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-zbvpf:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 49m default-scheduler Successfully assigned flyte/minio-799fc77896-6m85f to bea93849dc41
Normal Pulling 49m kubelet Pulling image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>"
Normal Pulled 47m kubelet Successfully pulled image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" in 1m56.1123313s
Normal Created 45m (x5 over 47m) kubelet Created container minio
Normal Started 45m (x5 over 47m) kubelet Started container minio
Normal Pulled 9m (x12 over 46m) kubelet Container image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" already present on machine
Warning BackOff 4m6s (x187 over 46m) kubelet Back-off restarting failed container
glamorous-carpet-83516
05/19/2022, 9:01 AMlate-garage-39497
05/19/2022, 9:05 AMkubectl logs minio-799fc77896-6m85f
, or any other pod I get that the pod is not found. Not sure if this the right command to use in k3sglamorous-carpet-83516
05/19/2022, 9:08 AM-n flyte
late-garage-39497
05/19/2022, 10:00 AM09:56:21.69
09:56:21.69 Welcome to the Bitnami minio container
09:56:21.70 Subscribe to project updates by watching <https://github.com/bitnami/bitnami-docker-minio>
09:56:21.70 Submit issues and feature requests at <https://github.com/bitnami/bitnami-docker-minio/issues>
09:56:21.70
09:56:21.70 INFO ==> ** Starting MinIO setup **
09:56:21.74 INFO ==> Starting MinIO in background...
09:57:02.62 INFO ==> Adding local Minio host to 'mc' configuration...
09:57:02.86 INFO ==> MinIO is already stopped...
glamorous-carpet-83516
05/19/2022, 11:09 AMlate-garage-39497
05/20/2022, 10:38 AMtall-lock-23197
freezing-airport-6809
tall-lock-23197
late-garage-39497
05/23/2022, 7:17 AMdig
command always return NXDOMAIN
. However if I specify a name server I get the correct response.
I've also cleaned my DNS cache .
Any ideas what could be wrong?tall-lock-23197
opta output
command give you four nameservers?late-garage-39497
05/23/2022, 12:38 PMtall-lock-23197
late-garage-39497
05/23/2022, 1:12 PMtall-lock-23197
late-garage-39497
05/23/2022, 1:44 PMNXDOMAIN
tall-lock-23197
icy-agent-73298
05/23/2022, 1:53 PMlate-garage-39497
05/23/2022, 1:55 PMicy-agent-73298
05/23/2022, 2:00 PMlate-garage-39497
05/24/2022, 9:17 AMicy-agent-73298
05/24/2022, 10:17 AMmost-sunset-30029
05/24/2022, 9:40 PM