https://flyte.org logo
Join Slack
Powered by
Hi all! :wave: I've started exploring flyte recent...
# announcements
l
Hi all! 👋 I've started exploring flyte recently but I don't seem to be able to run workflows on my sandbox environment. I've created a sandbox environment using the command 
flytectl sandbox start
and exported the env vars for kube and flyte configuration. I'm trying to run the hello world example from flytesnacks and I've created a virtual env using poetry, in which I installed pyflyte (1.0.2). I can run it locally no problem, but when I add the 
remote
flag, I get a connection reset by peer error. Any ideas what is wrong on my setup? I'm on WSL2
y
hi
googling WSL2
oh… windows.
can you 
ls ~/.flyte
and if there’s a config.yaml in there can you cat it?
also could you paste the full error?
@Marc Paquette
k
cc @Kevin Su / @Yuvraj do you folks know if WSL2 works?
l
Hi! @Yee I have a 
config-sandbox.yaml
which was created by the 
flytectl command
Copy code
Traceback (most recent call last):
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 443, in put_data
    DataPersistencePlugins.find_plugin(remote_path)().put(local_path, remote_path, recursive=is_multipart)
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/extras/persistence/http.py", line 67, in put
    rsp = requests.put(
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/api.py", line 132, in put
    return request('put', url, data=data, **kwargs)
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/clis/sdk_in_container/run.py", line 473, in _run
    wf = remote.register_script(
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/remote/remote.py", line 536, in register_script
    upload_location, md5_bytes = fast_register_single_script(
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/tools/script_mode.py", line 118, in fast_register_single_script
    flyte_ctx.file_access.put_data(archive_fname, upload_location.signed_url)
  File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 445, in put_data
    raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmpx5p7zdl2/script_mode.tar.gz to <http://localhost:30084/my-s3-bucket/gp/flytesnacks/development/ZVTXNVVEVCVG6EWW4ECBUKYBCU%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20220518%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220518T172729Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=458ac5b8322ade953b25890b321141ae04956eda640d11674cde26dc743b906a> (recursive=False).
Full Error here
k
It looks like flytekit can’t connect to minio. are you able to open flyte console (30081) in the browser?
l
the config:
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:30081
  #authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
I know see that I removed the authentication
k
@Luísa Braga Simões Could you help me check if the port is open or not by using 
netstat -anvp tcp | awk 'NR<3 || /LISTEN/
? port 30081, 30082, 30084 should be open.
l
Copy code
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:30080           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:30081           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:30082           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:30084           0.0.0.0:*               LISTEN
they seem to be
The minio pod seems to be in a crash loop 😕
Copy code
Containers:
  minio:
    Container ID:   <docker://b97239c7282e279b27ecdd7b59ccde6dfa9224cdee6581ba7d6d5362fc335ad>c
    Image:          <http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>
    Image ID:       <docker-pullable://ecr.flyte.org/bitnami/minio@sha256:547a4d0fdc82d5213fef3f4f7215fee788398238c5184a8555837bd3f649525e>
    Ports:          9000/TCP, 9001/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 19 May 2022 09:49:18 +0100
      Finished:     Thu, 19 May 2022 09:49:28 +0100
    Ready:          False
    Restart Count:  13
    Limits:
      cpu:     200m
      memory:  512Mi
    Requests:
      cpu:     10m
      memory:  128Mi
    Environment:
      MINIO_ACCESS_KEY:       minio
      MINIO_SECRET_KEY:       miniostorage
      MINIO_DEFAULT_BUCKETS:  my-s3-bucket
    Mounts:
      /data from minio-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zbvpf (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  minio-storage:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-zbvpf:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  49m                   default-scheduler  Successfully assigned flyte/minio-799fc77896-6m85f to bea93849dc41
  Normal   Pulling    49m                   kubelet            Pulling image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>"
  Normal   Pulled     47m                   kubelet            Successfully pulled image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" in 1m56.1123313s
  Normal   Created    45m (x5 over 47m)     kubelet            Created container minio
  Normal   Started    45m (x5 over 47m)     kubelet            Started container minio
  Normal   Pulled     9m (x12 over 46m)     kubelet            Container image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" already present on machine
  Warning  BackOff    4m6s (x187 over 46m)  kubelet            Back-off restarting failed container
k
is possible to see the log in minio pod?
l
I don't seem to be able to do it. With 
kubectl logs minio-799fc77896-6m85f
, or any other pod I get that the pod is not found. Not sure if this the right command to use in k3s
k
should add the namespace flag 
-n flyte
l
Copy code
09:56:21.69
 09:56:21.69 Welcome to the Bitnami minio container
 09:56:21.70 Subscribe to project updates by watching <https://github.com/bitnami/bitnami-docker-minio>
 09:56:21.70 Submit issues and feature requests at <https://github.com/bitnami/bitnami-docker-minio/issues>
 09:56:21.70
 09:56:21.70 INFO  ==> ** Starting MinIO setup **
 09:56:21.74 INFO  ==> Starting MinIO in background...
 09:57:02.62 INFO  ==> Adding local Minio host to 'mc' configuration...
 09:57:02.86 INFO  ==> MinIO is already stopped...
k
I’m going to test it on windows
👍 1
l
I'm now following this tutorial to run a cluster on my AWS account. However the link in DNS Delegation section seems to be broken. Can you help find me what I have to do in this step? Thanks 🙂
s
🙏 1
🙌 1
k
@Samhita Alla do you know what’s broken
s
The link’s broken. Will fix it.
❤️ 1
l
I'm still having issues on the DNS part. I have the hosted zone on my aws account, the name servers are there however the 
dig
command always return 
NXDOMAIN
. However if I specify a name server I get the correct response. I've also cleaned my DNS cache . Any ideas what could be wrong?
s
@Luísa Braga Simões, did 
opta output
command give you four nameservers?
l
yup it did, the same one defined in route53 (i'm deploying in aws)
s
Can you rerun opta deployment by deleting the route 53 hosted zones?
l
you mean deleting the hosted zone and re-run opta?
s
Yes, right from the DNS section.
l
just did, no changes there 😕 the dig command still return 
NXDOMAIN
s
cc: @Prafulla Mahindrakar
p
Hi @Luísa Braga Simões which domain registrar are you using like cloudfare. you would need to add those nameservers records for your dns.
l
I'm using Route53
p
l
How can I change the opta file to use a previously created hosted zone? Instead of having to register a new domain
p
adding @JD Palomino to give more info on the above question regarding opta
j
@Luísa Braga Simões ping me when you can zoom. It’ll be faster than back and forth messages
422 Views
Open in Slack
PreviousNext
Powered by