https://flyte.org logo
Join Slack
Powered by
Hi, community I just run ```flytectl demo start -...
# flyte-support
d
Hi, community I just run
Copy code
flytectl demo start --dev
Can anyone know why I have this message ? I've been waiting for a long time. This is my docker logs
Copy code
I0713 14:45:44.685281      61 controller.go:611] quota admission added evaluator for: leases.coordination.k8s.io
I0713 14:45:44.810411      61 topology_manager.go:200] "Topology Admit Handler"
I0713 14:45:44.810462      61 topology_manager.go:200] "Topology Admit Handler"
I0713 14:45:44.818907      61 topology_manager.go:200] "Topology Admit Handler"
I0713 14:45:44.823655      61 topology_manager.go:200] "Topology Admit Handler"
I0713 14:45:44.823737      61 topology_manager.go:200] "Topology Admit Handler"
I0713 14:45:44.825527      61 topology_manager.go:200] "Topology Admit Handler"
I0713 14:45:44.830225      61 topology_manager.go:200] "Topology Admit Handler"
I0713 14:45:44.983224      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qr7n\" (UniqueName: \"kubernetes.io/projected/a20ea692-ec90-49f5-b084-a08a028ce12c-kube-api-access-2qr7n\") pod \"coredns-b96499967-tkxnr\" (UID: \"a20ea692-ec90-49f5-b084-a08a028ce12c\") " pod="kube-system/coredns-b96499967-tkxnr"
I0713 14:45:44.983277      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ntgd\" (UniqueName: \"kubernetes.io/projected/f77f0e46-be4e-4e53-b739-a7a420137b95-kube-api-access-7ntgd\") pod \"flyte-sandbox-proxy-d95874857-5vtjd\" (UID: \"f77f0e46-be4e-4e53-b739-a7a420137b95\") " pod="flyte/flyte-sandbox-proxy-d95874857-5vtjd"
I0713 14:45:44.983298      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"custom-config-volume\" (UniqueName: \"kubernetes.io/configmap/a20ea692-ec90-49f5-b084-a08a028ce12c-custom-config-volume\") pod \"coredns-b96499967-tkxnr\" (UID: \"a20ea692-ec90-49f5-b084-a08a028ce12c\") " pod="kube-system/coredns-b96499967-tkxnr"
I0713 14:45:44.983320      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdstw\" (UniqueName: \"kubernetes.io/projected/5ff796b6-8a9a-451a-bce2-1fabad8918da-kube-api-access-vdstw\") pod \"flyte-sandbox-kubernetes-dashboard-6757db879c-mvsvv\" (UID: \"5ff796b6-8a9a-451a-bce2-1fabad8918da\") " pod="flyte/flyte-sandbox-kubernetes-dashboard-6757db879c-mvsvv"
I0713 14:45:44.983343      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e820f9d4-677e-4479-872e-8b216289099b-data\") pod \"flyte-sandbox-docker-registry-6df5565694-8rp5m\" (UID: \"e820f9d4-677e-4479-872e-8b216289099b\") " pod="flyte/flyte-sandbox-docker-registry-6df5565694-8rp5m"
I0713 14:45:44.983374      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24jrc\" (UniqueName: \"kubernetes.io/projected/c61420f1-367c-4207-a45f-52ea280d4fc6-kube-api-access-24jrc\") pod \"local-path-provisioner-7b7dc8d6f5-j52g4\" (UID: \"c61420f1-367c-4207-a45f-52ea280d4fc6\") " pod="kube-system/local-path-provisioner-7b7dc8d6f5-j52g4"
I0713 14:45:44.983392      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubernetes-dashboard-certs\" (UniqueName: \"kubernetes.io/secret/5ff796b6-8a9a-451a-bce2-1fabad8918da-kubernetes-dashboard-certs\") pod \"flyte-sandbox-kubernetes-dashboard-6757db879c-mvsvv\" (UID: \"5ff796b6-8a9a-451a-bce2-1fabad8918da\") " pod="flyte/flyte-sandbox-kubernetes-dashboard-6757db879c-mvsvv"
I0713 14:45:44.983442      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ff796b6-8a9a-451a-bce2-1fabad8918da-tmp-volume\") pod \"flyte-sandbox-kubernetes-dashboard-6757db879c-mvsvv\" (UID: \"5ff796b6-8a9a-451a-bce2-1fabad8918da\") " pod="flyte/flyte-sandbox-kubernetes-dashboard-6757db879c-mvsvv"
I0713 14:45:44.983529      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"flyte-sandbox-docker-registry-config\" (UniqueName: \"kubernetes.io/configmap/e820f9d4-677e-4479-872e-8b216289099b-flyte-sandbox-docker-registry-config\") pod \"flyte-sandbox-docker-registry-6df5565694-8rp5m\" (UID: \"e820f9d4-677e-4479-872e-8b216289099b\") " pod="flyte/flyte-sandbox-docker-registry-6df5565694-8rp5m"
I0713 14:45:44.983679      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbjwz\" (UniqueName: \"kubernetes.io/projected/e820f9d4-677e-4479-872e-8b216289099b-kube-api-access-rbjwz\") pod \"flyte-sandbox-docker-registry-6df5565694-8rp5m\" (UID: \"e820f9d4-677e-4479-872e-8b216289099b\") " pod="flyte/flyte-sandbox-docker-registry-6df5565694-8rp5m"
I0713 14:45:44.983720      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c61420f1-367c-4207-a45f-52ea280d4fc6-config-volume\") pod \"local-path-provisioner-7b7dc8d6f5-j52g4\" (UID: \"c61420f1-367c-4207-a45f-52ea280d4fc6\") " pod="kube-system/local-path-provisioner-7b7dc8d6f5-j52g4"
I0713 14:45:44.983734      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a20ea692-ec90-49f5-b084-a08a028ce12c-config-volume\") pod \"coredns-b96499967-tkxnr\" (UID: \"a20ea692-ec90-49f5-b084-a08a028ce12c\") " pod="kube-system/coredns-b96499967-tkxnr"
I0713 14:45:44.983791      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f77f0e46-be4e-4e53-b739-a7a420137b95-config\") pod \"flyte-sandbox-proxy-d95874857-5vtjd\" (UID: \"f77f0e46-be4e-4e53-b739-a7a420137b95\") " pod="flyte/flyte-sandbox-proxy-d95874857-5vtjd"
I0713 14:45:44.983810      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948f41ec-a739-4e55-8636-b88e42e76e2e-tmp-dir\") pod \"metrics-server-668d979685-6dvf9\" (UID: \"948f41ec-a739-4e55-8636-b88e42e76e2e\") " pod="kube-system/metrics-server-668d979685-6dvf9"
I0713 14:45:44.983840      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sns8\" (UniqueName: \"kubernetes.io/projected/948f41ec-a739-4e55-8636-b88e42e76e2e-kube-api-access-2sns8\") pod \"metrics-server-668d979685-6dvf9\" (UID: \"948f41ec-a739-4e55-8636-b88e42e76e2e\") " pod="kube-system/metrics-server-668d979685-6dvf9"
I0713 14:45:45.169529      61 node_lifecycle_controller.go:1192] Controller detected that some Nodes are Ready. Exiting master disruption mode.
E0713 14:45:45.206192      61 event_broadcaster.go:253] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"flyte-sandbox-postgresql-0.177174995c411b24", GenerateName:"", Namespace:"flyte", SelfLink:"", UID:"3bb90e0b-4884-4437-95f7-79427de4d434", ResourceVersion:"515", Generation:0, CreationTimestamp:time.Date(2023, time.July, 13, 14, 45, 37, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ZZZ_DeprecatedClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"k3s", Operation:"Update", APIVersion:"events.k8s.io/v1", Time:time.Date(2023, time.July, 13, 14, 45, 37, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0073b2060), Subresource:""}}}, EventTime:time.Date(2023, time.July, 13, 14, 45, 37, 267230916, time.Local), Series:(*v1.EventSeries)(0xc00605d340), ReportingController:"default-scheduler", ReportingInstance:"default-scheduler-99636619f4b2", Action:"Scheduling", Reason:"FailedScheduling", Regarding:v1.ObjectReference{Kind:"Pod", Namespace:"flyte", Name:"flyte-sandbox-postgresql-0", UID:"fe18443f-7779-4cd1-8822-e69534b74fca", APIVersion:"v1", ResourceVersion:"502", FieldPath:""}, Related:(*v1.ObjectReference)(nil), Note:"0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.", Type:"Warning", DeprecatedSource:v1.EventSource{Component:"", Host:""}, DeprecatedFirstTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeprecatedLastTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeprecatedCount:0}': 'Event "flyte-sandbox-postgresql-0.177174995c411b24" is invalid: [series.count: Invalid value: "": should be at least 2, eventTime: Invalid value: 2023-07-13 14:45:37.26723 +0000 UTC: field is immutable]' (will not retry!)
I0713 14:45:46.211777      61 request.go:601] Waited for 1.126210688s due to client-side throttling, not priority and fairness, request: POST:<https://127.0.0.1:6443/api/v1/namespaces/flyte/serviceaccounts/flyte-sandbox-kubernetes-dashboard/token>
I0713 14:45:47.299538      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"flyte-sandbox-minio-storage\" (UniqueName: \"kubernetes.io/host-path/69014e03-6f38-49ef-b37e-653cc71766b2-flyte-sandbox-minio-storage\") pod \"flyte-sandbox-minio-645c8ddf7c-2lqzz\" (UID: \"69014e03-6f38-49ef-b37e-653cc71766b2\") " pod="flyte/flyte-sandbox-minio-645c8ddf7c-2lqzz"
I0713 14:45:47.299570      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxqb\" (UniqueName: \"kubernetes.io/projected/69014e03-6f38-49ef-b37e-653cc71766b2-kube-api-access-bnxqb\") pod \"flyte-sandbox-minio-645c8ddf7c-2lqzz\" (UID: \"69014e03-6f38-49ef-b37e-653cc71766b2\") " pod="flyte/flyte-sandbox-minio-645c8ddf7c-2lqzz"
I0713 14:46:00.213730      61 topology_manager.go:200] "Topology Admit Handler"
I0713 14:46:00.284583      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"flyte-sandbox-db-storage\" (UniqueName: \"kubernetes.io/host-path/fe18443f-7779-4cd1-8822-e69534b74fca-flyte-sandbox-db-storage\") pod \"flyte-sandbox-postgresql-0\" (UID: \"fe18443f-7779-4cd1-8822-e69534b74fca\") " pod="flyte/flyte-sandbox-postgresql-0"
I0713 14:46:00.284613      61 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8ftq\" (UniqueName: \"kubernetes.io/projected/fe18443f-7779-4cd1-8822-e69534b74fca-kube-api-access-d8ftq\") pod \"flyte-sandbox-postgresql-0\" (UID: \"fe18443f-7779-4cd1-8822-e69534b74fca\") " pod="flyte/flyte-sandbox-postgresql-0"
E0713 14:46:00.424506      61 resource_quota_controller.go:413] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
W0713 14:46:00.893544      61 garbagecollector.go:747] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
f
do you see anything if you run “kubectl get pods -A”
d
f
looks like kubeconfig wasn’t setup
try “docker exec flyte-sandbox kubectl get pods -A”
d
It works
Should I setup my kubeconfig ?
f
hmm flytectl should’ve done it for you
flytectl demo start —dev still hasn’t exited?
d
Haven;t exited
I just follow the comand above
and docker ps -a just tell me that it starts for a long while
f
any file in “~/.flyte/sandbox”?
d
yes
Should I delete my kubeconfig ?
f
set the KUBECONFIG env var to that and try “kubectl get pods -A”?
you can set it inline
d
export KUBECONFIG=~/.flyte/sandbox/kubeconfig
do you mean like this ?
👍 1
now I have next error, but we make a progress !
f
no that just means it’s not the right kubeconfig
you can “unset KUBECONFIG”
i’ll have to see flytectl code to see where it failed
is it still stuck in waiting for server?
d
I have reinstall flytectl due to my env problem, maybe I have triogger some edge case ?
yes the server is keep running
would you like to google meet ?
I can share my screen
Copy code
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyT0RreU5qQTBOVFF3SGhjTk1qTXdOekV6TVRVd01EVTBXaGNOTXpNd056RXdNVFV3TURVMApXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyT0RreU5qQTBOVFF3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFTNDVncy9udWozRzErYVpQekxsSFhnNjJqKzJuV3FUYndVZTNWZGdFZXoKMTNMT0sxWnUzU1g3Z2xNNUpNRW1sNFArdEIwa3FZaEhQU0VHdkU5RWpyaXFvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVTZ3QTI3Yy9QdEVmVnRnNU4zQ3pzCkhJMU1VQzR3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnRFl5NnF2U1JGdWdpZ0xKelgwR1pEU2xXbEhlYzYxMzkKRVpiMWtoTlRBZllDSVFEbk9kcGN5ejZ5Z29yRU5vaThDQkxEaURLSVI1QTByUUxvdEF0OW9rSTkyZz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: <https://127.0.0.1:6443>
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRlZ0F3SUJBZ0lJZnFmTC9wTHQ3aGt3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOamc1TWpZd05EVTBNQjRYRFRJek1EY3hNekUxTURBMU5Gb1hEVEkwTURjeApNakUxTURBMU5Gb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJCbElSOS9vWFJoaGZTWXUKaE1wRG93SSsxTlJhYUh1bDVFVHJjdjI0VFVURFdUT2t3aE9TYmI4QkVaTnh6bXprWmdWYnM4aWdEbThMNnV2RgpIaDcyUEIyalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCUW1hdWxFeGkzdkxyWTQ4NmFOTXEzWWlTbGxoakFLQmdncWhrak9QUVFEQWdOSUFEQkYKQWlFQXJUdFZ5b255L2VMRW0wSkhIaEExZnVVV25IQ3E0czdQdlZuR0loNFIxWTBDSUhyS3YrclVPejhVNFdWYwpXMldrbnpya2VzN3IyQ1p0QnNtK29Pbys2TlBzCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdFkyeHAKWlc1MExXTmhRREUyT0RreU5qQTBOVFF3SGhjTk1qTXdOekV6TVRVd01EVTBXaGNOTXpNd056RXdNVFV3TURVMApXakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwWlc1MExXTmhRREUyT0RreU5qQTBOVFF3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFUWmtrZysyeWJvZHVmZnUyaEtPb08vK1loemdiTWlNMHdjdDhOZEx6c3AKbUlOdjJsdW44VWRhaG1mZEhLY2lXcitQQmZib01IUVpHWnpVQ2p3OTlSaXZvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVUptcnBSTVl0N3k2Mk9QT21qVEt0CjJJa3BaWVl3Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQU44RGkrWjhHUS9yanFiNzllOGR4b0pZMUEzazBEZkcKYjNsemlTOWRKMThjQWlFQTdEdHZkanU4UFQ0NzVjN2dURmpJZWNEOHdCcDcrMUJpejIvTHh6RmQ3VGM9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUFUWTZja2I3R2IxNTRVNW9DcWdmeHdLbE9EVHU4NVZvZEtaY0hwaHh6WC9vQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFR1VoSDMraGRHR0Y5Smk2RXlrT2pBajdVMUZwb2U2WGtST3R5L2JoTlJNTlpNNlRDRTVKdAp2d0VSazNIT2JPUm1CVnV6eUtBT2J3dnE2OFVlSHZZOEhRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=
This is my kubeconfig
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:30080
  authType: Pkce
  insecure: true
console:
  endpoint: <http://localhost:30080>
logger:
  show-source: true
  level: 0
f
d
This is my ~/.flyte/config-sandbox.yaml
f
let me get back to you
d
Sure, thank you
@freezing-boots-56761 I have work tomorrow, so I have to sleep now. I will try your solution and reply you tomorrow in UTF+8 time zone.
Thank you very much
export FLYTECTL_CONFIG=~/.flyte/config-sandbox.yaml export KUBECONFIG=~/.flyte/sandbox/kubeconfig
My bashrc in ubuntu 22 use these env variables
Hope these help
f
oh hmm, can you drop those env vars?
d
yes syre'
yes
f
shouldn’t be necessary. then teardown and recreate
d
It doesn;t work
I have teared down
f
did you unset in current shell?
d
Is this what you want ?
👍 1
It still doesn't work
😞 1
Thanks for your time, maybe I will try it tommorow
f
i’ll try to repro on my end
d
Sure, if success, please share your process.
Thank you really really much
docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE cr.flyte.org/flyteorg/flyte-sandbox-bundled sha-951a93080972bbcf85bc92ec54ac62e074be887f b66458f6f9b9 41 hours ago 1.49GB
This is the image I use, though it might not help
f
i tried with the same. couldnt repro on mac or linux 😞
d
So you work well?
Can you guide me how to uninstall flyte totally?
Maybe it will help
When I reinstall it again
f
maybe you can just 
rm -rf ~/.flyte
and try again
but we can follow-up tomorrow
gratitude thank you 2
d
Sure
Hope you are available tomorrow this time
👍 1
60 Views
Open in Slack
PreviousNext
Powered by