Hi all wave I ve started exploring flyte recently but I don

Question

Hi all wave I ve started exploring flyte recently but I don t seem to be able to run workflows on my sandbox environment I ve created a sandbox environment using the command `flytectl sandbox start` and exported the env vars for kube and flyte configuration I m trying to run the hello world example from <https github com flyteorg flytesnacks|flytesnacks> and I ve created a virtual env using poetry in which I installed pyflyte 1 0 2 I can run it locally no problem but when I add the `remote` flag I get a connection reset by peer error Any ideas what is wrong on my setup I m on WSL2

Yee · Answer

hi

Yee · Answer

googling WSL2

Yee · Answer

oh windows

Yee · Answer

can you `ls ~ flyte` and if there s a config yaml in there can you cat it

Yee · Answer

also could you paste the full error

Yee · Answer

< Marc Paquette>

Ketan (kumare3) · Answer

cc < Kevin Su> < Yuvraj> do you folks know if WSL2 works

Luísa Braga Simões · Answer

Hi < Yee> I have a `config sandbox yaml` which was created by the `flytectl command`

Luísa Braga Simões · Answer

```Traceback most recent call last File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages flytekit core data persistence py line 443 in put data DataPersistencePlugins find plugin remote path put local path remote path recursive=is multipart File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages flytekit extras persistence http py line 67 in put rsp = requests put File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages requests api py line 132 in put return request put url data=data kwargs File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages requests api py line 61 in request return session request method=method url=url kwargs File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages requests sessions py line 529 in request resp = self send prep send kwargs File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages requests sessions py line 645 in send r = adapter send request kwargs File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages requests adapters py line 501 in send raise ConnectionError err request=request requests exceptions ConnectionError Connection aborted ConnectionResetError 104 Connection reset by peer The above exception was the direct cause of the following exception Traceback most recent call last File home luisacabs exploring flyte flytesnacks cookbook venv bin pyflyte line 8 in lt module gt sys exit main File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages click core py line 1130 in call return self main args kwargs File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages click core py line 1055 in main rv = self invoke ctx File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages click core py line 1657 in invoke return process result sub ctx command invoke sub ctx File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages click core py line 1657 in invoke return process result sub ctx command invoke sub ctx File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages click core py line 1657 in invoke return process result sub ctx command invoke sub ctx File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages click core py line 1404 in invoke return ctx invoke self callback ctx params File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages click core py line 760 in invoke return callback args kwargs File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages flytekit clis sdk in container run py line 473 in run wf = remote register script File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages flytekit remote remote py line 536 in register script upload location md5 bytes = fast register single script File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages flytekit tools script mode py line 118 in fast register single script flyte ctx file access put data archive fname upload location signed url File home luisacabs exploring flyte flytesnacks cookbook venv lib python3 8 site packages flytekit core data persistence py line 445 in put data raise FlyteAssertion flytekit exceptions user FlyteAssertion Failed to put data from tmp tmpx5p7zdl2 script mode tar gz to <http localhost 30084 my s3 bucket gp flytesnacks development ZVTXNVVEVCVG6EWW4ECBUKYBCU 3D 3D 3D 3D 3D 3D scriptmode tar gz X Amz Algorithm=AWS4 HMAC SHA256 amp X Amz Credential=minio 2F20220518 2Fus east 1 2Fs3 2Faws4 request amp X Amz Date=20220518T172729Z amp X Amz Expires=3600 amp X Amz SignedHeaders=content md5 3Bhost amp X Amz Signature=458ac5b8322ade953b25890b321141ae04956eda640d11674cde26dc743b906a> recursive=False ``` Full Error here

Kevin Su · Answer

It looks like flytekit can t connect to minio are you able to open flyte console 30081 in the browser

Luísa Braga Simões · Answer

the config ```admin For GRPC endpoints you might want to use dns flyte myexample com endpoint localhost 30081 authType Pkce insecure true logger show source true level 0``` I know see that I removed the authentication

Kevin Su · Answer

< Luísa Braga Simões> Could you help me check if the port is open or not by using `netstat anvp tcp | awk NR lt 3 || LISTEN ` port 30081 30082 30084 should be open

Luísa Braga Simões · Answer

```Proto Recv Q Send Q Local Address Foreign Address State tcp 0 0 0 0 0 0 30080 0 0 0 0 LISTEN tcp 0 0 0 0 0 0 30081 0 0 0 0 LISTEN tcp 0 0 0 0 0 0 30082 0 0 0 0 LISTEN tcp 0 0 0 0 0 0 30084 0 0 0 0 LISTEN``` they seem to be

Luísa Braga Simões · Answer

The minio pod seems to be in a crash loop confused

Luísa Braga Simões · Answer

```Containers minio Container ID c Image Image ID Ports 9000 TCP 9001 TCP Host Ports 0 TCP 0 TCP State Waiting Reason CrashLoopBackOff Last State Terminated Reason Error Exit Code 1 Started Thu 19 May 2022 09 49 18 +0100 Finished Thu 19 May 2022 09 49 28 +0100 Ready False Restart Count 13 Limits cpu 200m memory 512Mi Requests cpu 10m memory 128Mi Environment MINIO ACCESS KEY minio MINIO SECRET KEY miniostorage MINIO DEFAULT BUCKETS my s3 bucket Mounts data from minio storage rw var run secrets kubernetes io serviceaccount from kube api access zbvpf ro Conditions Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes minio storage Type EmptyDir a temporary directory that shares a pod s lifetime Medium SizeLimit lt unset gt kube api access zbvpf Type Projected a volume that contains injected data from multiple sources TokenExpirationSeconds 3607 ConfigMapName kube root ca crt ConfigMapOptional lt nil gt DownwardAPI true QoS Class Burstable Node Selectors lt none gt Tolerations op=Exists for 300s op=Exists for 300s Events Type Reason Age From Message Normal Scheduled 49m default scheduler Successfully assigned flyte minio 799fc77896 6m85f to bea93849dc41 Normal Pulling 49m kubelet Pulling image Normal Pulled 47m kubelet Successfully pulled image in 1m56 1123313s Normal Created 45m x5 over 47m kubelet Created container minio Normal Started 45m x5 over 47m kubelet Started container minio Normal Pulled 9m x12 over 46m kubelet Container image already present on machine Warning BackOff 4m6s x187 over 46m kubelet Back off restarting failed container```

Kevin Su · Answer

is possible to see the log in minio pod

Luísa Braga Simões · Answer

I don t seem to be able to do it With `kubectl logs minio 799fc77896 6m85f` or any other pod I get that the pod is not found Not sure if this the right command to use in k3s

Kevin Su · Answer

should add the namespace flag ` n flyte`

Luísa Braga Simões · Answer

``` 09 56 21 69 09 56 21 69 Welcome to the Bitnami minio container 09 56 21 70 Subscribe to project updates by watching 09 56 21 70 Submit issues and feature requests at 09 56 21 70 09 56 21 70 INFO == gt Starting MinIO setup 09 56 21 74 INFO == gt Starting MinIO in background 09 57 02 62 INFO == gt Adding local Minio host to mc configuration 09 57 02 86 INFO == gt MinIO is already stopped ```

Kevin Su · Answer

I m going to test it on windows

Luísa Braga Simões · Answer

I m now following <https docs flyte org en latest deployment aws opta html|this tutorial> to run a cluster on my AWS account However the link in DNS Delegation section seems to be broken Can you help find me what I have to do in this step Thanks slightly smiling face

Samhita Alla · Answer

I think you can follow <https docs opta dev features dns and cert dns> guide

Ketan (kumare3) · Answer

< Samhita Alla> do you know what s broken

Samhita Alla · Answer

The link s broken Will fix it

Luísa Braga Simões · Answer

I m still having issues on the DNS part I have the hosted zone on my aws account the name servers are there however the `dig` command always return `NXDOMAIN` However if I specify a name server I get the correct response I ve also cleaned my DNS cache Any ideas what could be wrong

Samhita Alla · Answer

< Luísa Braga Simões> did `opta output` command give you four nameservers

Luísa Braga Simões · Answer

yup it did the same one defined in route53 i m deploying in aws

Samhita Alla · Answer

Can you rerun opta deployment by deleting the route 53 hosted zones

Luísa Braga Simões · Answer

you mean deleting the hosted zone and re run opta

Samhita Alla · Answer

Yes right from the DNS section

Luísa Braga Simões · Answer

just did no changes there confused the dig command still return `NXDOMAIN`

Samhita Alla · Answer

cc < Prafulla Mahindrakar>

Prafulla Mahindrakar · Answer

Hi < Luísa Braga Simões> which domain registrar are you using like cloudfare you would need to add those nameservers records for your dns

Luísa Braga Simões · Answer

I m using Route53

Prafulla Mahindrakar · Answer

Can you try this trobleshooting guide from aws <https aws amazon com premiumsupport knowledge center route 53 troubleshoot nxdomain responses >

Luísa Braga Simões · Answer

How can I change the opta file to use a previously created hosted zone Instead of having to register a new domain

Prafulla Mahindrakar · Answer

adding < JD Palomino> to give more info on the above question regarding opta

JD Palomino · Answer

< Luísa Braga Simões> ping me when you can zoom It ll be faster than back and forth messages