Thread
#announcements
    Luísa Braga Simões

    Luísa Braga Simões

    4 months ago
    Hi all! 👋 I've started exploring flyte recently but I don't seem to be able to run workflows on my sandbox environment. I've created a sandbox environment using the command
    flytectl sandbox start
    and exported the env vars for kube and flyte configuration. I'm trying to run the hello world example from flytesnacks and I've created a virtual env using poetry, in which I installed pyflyte (1.0.2). I can run it locally no problem, but when I add the
    remote
    flag, I get a connection reset by peer error. Any ideas what is wrong on my setup? I'm on WSL2
    Yee

    Yee

    4 months ago
    hi
    googling WSL2
    oh… windows.
    can you
    ls ~/.flyte
    and if there’s a config.yaml in there can you cat it?
    also could you paste the full error?
    @Marc Paquette
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    cc @Kevin Su / @Yuvraj do you folks know if WSL2 works?
    Luísa Braga Simões

    Luísa Braga Simões

    4 months ago
    Hi! @Yee I have a
    config-sandbox.yaml
    which was created by the
    flytectl command
    Traceback (most recent call last):
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 443, in put_data
        DataPersistencePlugins.find_plugin(remote_path)().put(local_path, remote_path, recursive=is_multipart)
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/extras/persistence/http.py", line 67, in put
        rsp = requests.put(
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/api.py", line 132, in put
        return request('put', url, data=data, **kwargs)
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/api.py", line 61, in request
        return session.request(method=method, url=url, **kwargs)
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
        resp = self.send(prep, **send_kwargs)
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
        r = adapter.send(request, **kwargs)
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/requests/adapters.py", line 501, in send
        raise ConnectionError(err, request=request)
    requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/bin/pyflyte", line 8, in <module>
        sys.exit(main())
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/clis/sdk_in_container/run.py", line 473, in _run
        wf = remote.register_script(
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/remote/remote.py", line 536, in register_script
        upload_location, md5_bytes = fast_register_single_script(
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/tools/script_mode.py", line 118, in fast_register_single_script
        flyte_ctx.file_access.put_data(archive_fname, upload_location.signed_url)
      File "/home/luisacabs/exploring/flyte/flytesnacks/cookbook/.venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 445, in put_data
        raise FlyteAssertion(
    flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmpx5p7zdl2/script_mode.tar.gz to <http://localhost:30084/my-s3-bucket/gp/flytesnacks/development/ZVTXNVVEVCVG6EWW4ECBUKYBCU%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20220518%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220518T172729Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=458ac5b8322ade953b25890b321141ae04956eda640d11674cde26dc743b906a> (recursive=False).
    Full Error here
    Kevin Su

    Kevin Su

    4 months ago
    It looks like flytekit can’t connect to minio. are you able to open flyte console (30081) in the browser?
    Luísa Braga Simões

    Luísa Braga Simões

    4 months ago
    the config:
    admin:
      # For GRPC endpoints you might want to use dns:///flyte.myexample.com
      endpoint: localhost:30081
      #authType: Pkce
      insecure: true
    logger:
      show-source: true
      level: 0
    I know see that I removed the authentication
    Kevin Su

    Kevin Su

    4 months ago
    @Luísa Braga Simões Could you help me check if the port is open or not by using
    netstat -anvp tcp | awk 'NR<3 || /LISTEN/
    ? port 30081, 30082, 30084 should be open.
    Luísa Braga Simões

    Luísa Braga Simões

    4 months ago
    Proto Recv-Q Send-Q Local Address           Foreign Address         State
    tcp        0      0 0.0.0.0:30080           0.0.0.0:*               LISTEN
    tcp        0      0 0.0.0.0:30081           0.0.0.0:*               LISTEN
    tcp        0      0 0.0.0.0:30082           0.0.0.0:*               LISTEN
    tcp        0      0 0.0.0.0:30084           0.0.0.0:*               LISTEN
    they seem to be
    The minio pod seems to be in a crash loop 😕
    Containers:
      minio:
        Container ID:   <docker://b97239c7282e279b27ecdd7b59ccde6dfa9224cdee6581ba7d6d5362fc335ad>c
        Image:          <http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>
        Image ID:       <docker-pullable://ecr.flyte.org/bitnami/minio@sha256:547a4d0fdc82d5213fef3f4f7215fee788398238c5184a8555837bd3f649525e>
        Ports:          9000/TCP, 9001/TCP
        Host Ports:     0/TCP, 0/TCP
        State:          Waiting
          Reason:       CrashLoopBackOff
        Last State:     Terminated
          Reason:       Error
          Exit Code:    1
          Started:      Thu, 19 May 2022 09:49:18 +0100
          Finished:     Thu, 19 May 2022 09:49:28 +0100
        Ready:          False
        Restart Count:  13
        Limits:
          cpu:     200m
          memory:  512Mi
        Requests:
          cpu:     10m
          memory:  128Mi
        Environment:
          MINIO_ACCESS_KEY:       minio
          MINIO_SECRET_KEY:       miniostorage
          MINIO_DEFAULT_BUCKETS:  my-s3-bucket
        Mounts:
          /data from minio-storage (rw)
          /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zbvpf (ro)
    Conditions:
      Type              Status
      Initialized       True 
      Ready             False 
      ContainersReady   False 
      PodScheduled      True 
    Volumes:
      minio-storage:
        Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
        Medium:     
        SizeLimit:  <unset>
      kube-api-access-zbvpf:
        Type:                    Projected (a volume that contains injected data from multiple sources)
        TokenExpirationSeconds:  3607
        ConfigMapName:           kube-root-ca.crt
        ConfigMapOptional:       <nil>
        DownwardAPI:             true
    QoS Class:                   Burstable
    Node-Selectors:              <none>
    Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                                 <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
    Events:
      Type     Reason     Age                   From               Message
      ----     ------     ----                  ----               -------
      Normal   Scheduled  49m                   default-scheduler  Successfully assigned flyte/minio-799fc77896-6m85f to bea93849dc41
      Normal   Pulling    49m                   kubelet            Pulling image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>"
      Normal   Pulled     47m                   kubelet            Successfully pulled image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" in 1m56.1123313s
      Normal   Created    45m (x5 over 47m)     kubelet            Created container minio
      Normal   Started    45m (x5 over 47m)     kubelet            Started container minio
      Normal   Pulled     9m (x12 over 46m)     kubelet            Container image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" already present on machine
      Warning  BackOff    4m6s (x187 over 46m)  kubelet            Back-off restarting failed container
    Kevin Su

    Kevin Su

    4 months ago
    is possible to see the log in minio pod?
    Luísa Braga Simões

    Luísa Braga Simões

    4 months ago
    I don't seem to be able to do it. With
    kubectl logs minio-799fc77896-6m85f
    , or any other pod I get that the pod is not found. Not sure if this the right command to use in k3s
    Kevin Su

    Kevin Su

    4 months ago
    should add the namespace flag
    -n flyte
    Luísa Braga Simões

    Luísa Braga Simões

    4 months ago
    09:56:21.69
     09:56:21.69 Welcome to the Bitnami minio container
     09:56:21.70 Subscribe to project updates by watching <https://github.com/bitnami/bitnami-docker-minio>
     09:56:21.70 Submit issues and feature requests at <https://github.com/bitnami/bitnami-docker-minio/issues>
     09:56:21.70
     09:56:21.70 INFO  ==> ** Starting MinIO setup **
     09:56:21.74 INFO  ==> Starting MinIO in background...
     09:57:02.62 INFO  ==> Adding local Minio host to 'mc' configuration...
     09:57:02.86 INFO  ==> MinIO is already stopped...
    Kevin Su

    Kevin Su

    4 months ago
    I’m going to test it on windows
    Luísa Braga Simões

    Luísa Braga Simões

    4 months ago
    I'm now following this tutorial to run a cluster on my AWS account. However the link in DNS Delegation section seems to be broken. Can you help find me what I have to do in this step? Thanks 🙂
    Samhita Alla

    Samhita Alla

    4 months ago
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    @Samhita Alla do you know what’s broken
    Samhita Alla

    Samhita Alla

    4 months ago
    The link’s broken. Will fix it.
    Luísa Braga Simões

    Luísa Braga Simões

    4 months ago
    I'm still having issues on the DNS part. I have the hosted zone on my aws account, the name servers are there however the
    dig
    command always return
    NXDOMAIN
    . However if I specify a name server I get the correct response. I've also cleaned my DNS cache . Any ideas what could be wrong?
    Samhita Alla

    Samhita Alla

    4 months ago
    @Luísa Braga Simões, did
    opta output
    command give you four nameservers?
    Luísa Braga Simões

    Luísa Braga Simões

    4 months ago
    yup it did, the same one defined in route53 (i'm deploying in aws)
    Samhita Alla

    Samhita Alla

    4 months ago
    Can you rerun opta deployment by deleting the route 53 hosted zones?
    Luísa Braga Simões

    Luísa Braga Simões

    4 months ago
    you mean deleting the hosted zone and re-run opta?
    Samhita Alla

    Samhita Alla

    4 months ago
    Yes, right from the DNS section.
    Luísa Braga Simões

    Luísa Braga Simões

    4 months ago
    just did, no changes there 😕 the dig command still return
    NXDOMAIN
    Samhita Alla

    Samhita Alla

    4 months ago
    cc: @Prafulla Mahindrakar
    p

    Prafulla Mahindrakar

    4 months ago
    Hi @Luísa Braga Simões which domain registrar are you using like cloudfare. you would need to add those nameservers records for your dns.
    Luísa Braga Simões

    Luísa Braga Simões

    4 months ago
    I'm using Route53
    p

    Prafulla Mahindrakar

    4 months ago
    Luísa Braga Simões

    Luísa Braga Simões

    4 months ago
    How can I change the opta file to use a previously created hosted zone? Instead of having to register a new domain
    p

    Prafulla Mahindrakar

    4 months ago
    adding @JD Palomino to give more info on the above question regarding opta
    j

    JD Palomino

    4 months ago
    @Luísa Braga Simões ping me when you can zoom. It’ll be faster than back and forth messages