Blair Anson
04/20/2023, 6:48 AMflytectl create project
command?Kevin Su
04/20/2023, 7:21 AMBlair Anson
04/20/2023, 7:29 AM$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
flyte flyte-backend-flyte-binary-786cf7cdf9-vpjt6 1/1 Running 0 141m
flytesnacks-development ff4da8998ef014132b54-n0-0 0/1 Completed 0 66s
flytesnacks-development ff4da8998ef014132b54-n1-0 0/1 Completed 0 66s
flytesnacks-development ff4da8998ef014132b54-n2-0 0/1 Completed 0 66s
kube-system aws-node-c4x5t 1/1 Running 0 5d1h
kube-system aws-node-c6q29 1/1 Running 0 5d1h
kube-system aws-node-tw6n2 1/1 Running 0 5d1h
kube-system coredns-657694c6f4-7cbdr 1/1 Running 0 5d1h
kube-system coredns-657694c6f4-nxhpd 1/1 Running 0 5d1h
kube-system ebs-csi-controller-c869dc55b-bfqgm 6/6 Running 0 5d1h
kube-system ebs-csi-controller-c869dc55b-q4nb7 6/6 Running 0 5d1h
kube-system ebs-csi-node-ht6hr 3/3 Running 0 5d1h
kube-system ebs-csi-node-lzmkv 3/3 Running 0 5d1h
kube-system ebs-csi-node-x6pxr 3/3 Running 0 5d1h
kube-system efs-csi-controller-759b88757-cb97b 3/3 Running 0 23h
kube-system efs-csi-controller-759b88757-dxv2x 3/3 Running 0 23h
kube-system efs-csi-node-2w8hg 3/3 Running 0 23h
kube-system efs-csi-node-552xm 3/3 Running 0 23h
kube-system efs-csi-node-7fhlg 3/3 Running 0 23h
kube-system kube-proxy-7fdnj 1/1 Running 0 5d1h
kube-system kube-proxy-rfjck 1/1 Running 0 5d1h
kube-system kube-proxy-rx2mx 1/1 Running 0 5d1h
If I look in flyte-backend-flyte-binary
all I see are SQL statements from pod startup
2023/04/20 05:05:35 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/finisher_api.go:509
[0.462ms] [rows:1] SELECT count(*) FROM pg_indexes WHERE tablename = 'artifacts' AND indexname = 'artifacts_dataset_uuid_idx' AND schemaname = CURRENT_SCHEMA()
{"metrics-prefix":"flyte:","certDir":"/var/run/flyte/certs","localCert":true,"listenPort":9443,"serviceName":"flyte-backend-flyte-binary-webhook","servicePort":443,"secretName":"flyte-backend-flyte-binary-webhook-secret","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"gcpSecretManager":{"sidecarImage":"<http://gcr.io/google.com/cloudsdktool/cloud-sdk:alpine|gcr.io/google.com/cloudsdktool/cloud-sdk:alpine>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2"}}
If I look in the task pods all I see is this...
$ kubectl logs ff4da8998ef014132b54-n0-0 -n flytesnacks-development
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-vkny2wcs because the default path (/home/flytekit/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
tar: Removing leading `/' from member names
Is there somewhere else I should be looking, or is there a Log config I am meant to set?Kevin Su
04/20/2023, 7:40 AMBlair Anson
04/20/2023, 7:57 AMDan Rammer (hamersaw)
04/20/2023, 2:51 PMcluster_resource_manager
in FlyteAdmin to create a PV - this is not being created?
And then you're expecting Flyte to automatically detect the existence of the PV and mount it in k8s Pods? Or are you doing anything specific to mount?
Can you link to any docs you've used for this?PodTemplate
setup to automatically mount the volume for each task.Blair Anson
04/20/2023, 3:17 PMDan Rammer (hamersaw)
04/21/2023, 2:49 PMPods
you provided, the one you are looking at logs for is flyte-backend-flyte-binary-786cf7cdf9-vpjt6
. This is all of Flyte compiled into a single binary - so specifically you will be searching for flyteadmin logs regarding the cluster resource manager.Blair Anson
04/22/2023, 7:24 AM