Hi everyone, I wanted to test something basic and I wanted to leverage the local demo Flyte cluster for this: https://docs.flyte.org/en/latest/user_guide/environment_setup.html But for me even the toy example doesn't work with the demo cluster. I have started the demo cluster, exported FLYTECTL_CONFIG and got the following error:

pyflyte run --remote hello_world.py hello_world_wf
Running Execution on Remote.
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNIMPLEMENTED
        details: 
        Debug string UNKNOWN:Error received from peer  {grpc_message:"", grpc_status:12, created_time:"2024-03-01T10:46:26.305551071+01:00"}

Can you suggest something to make it work?

could you share the config in

~/.flyte/config-sandbox.yaml

?

sure admin: # For GRPC endpoints you might want to use dns:///flyte.myexample.com endpoint: localhost:30080 insecure: true # This is not a needed configuration, only useful if you want to explore the data in sandbox. For non sandbox, please # do not use this configuration, instead prefer to use aws, gcs, azure sessions. Flytekit, should use fsspec to # auto select the right backend to pull data as long as the sessions are configured. For Sandbox, this is special, as # minio is s3 compatible and we ship with minio in sandbox. storage: connection: endpoint: http://localhost:30002 access-key: minio secret-key: miniostorage

have you tried tearing down your cluster and restarting it?

flytectl demo teardown

and

flytectl demo start

I just did it and I am facing the same error

what's the output of

kubectl get pods -n flyte

?

yes,

NAME                                                  READY   STATUS    RESTARTS       AGE
flyte-sandbox-docker-registry-57789f56d-27zcw         1/1     Running   0              141m
flyte-sandbox-kubernetes-dashboard-6757db879c-nkmgs   1/1     Running   0              141m
flyte-sandbox-postgresql-0                            1/1     Running   0              141m
flyte-sandbox-buildkit-675bff7bd9-l58vv               1/1     Running   0              141m
flyte-sandbox-minio-85dc5446b6-lw9f7                  1/1     Running   0              141m
flyteagent-b98cfb65f-2dxjb                            1/1     Running   0              141m
flyte-sandbox-proxy-8f4f7fd9b-hpvl4                   1/1     Running   0              141m
flyte-sandbox-7f89d97699-67bwc                        1/1     Running   2 (140m ago)   141m

@average-finland-92144 i'm running out of ways to debug this. any idea what might be causing this issue?

@bored-needle-72209 could you please install

grpcurl

to validate any possible connection error to the gRPC server:

grpcurl --plaintext -v localhost:30080 list

It should output

flyteidl.service.AdminService
flyteidl.service.DataProxyService
flyteidl.service.SignalService
grpc.health.v1.Health
grpc.reflection.v1.ServerReflection
grpc.reflection.v1alpha.ServerReflection

You are right, I think the problem was that `30080`port was occupied. Is there some way to customize this port (to use 30081 port instead) in the demo cluster?

I think there were discussions about that in the past but no straightforward mechanism to do so. I tried invoking a config file as part of the

flytectl demo start -c <config>

command but it gets rewritten

I see, thank you for your help!