Hi, when using flyte-binary how can I view the fly...
# ask-the-community
b
Hi, when using flyte-binary how can I view the flyte-propeller logs to see any errors with workflow execution? And what about project creation, are there logs on the server showing changes done as the result of a
flytectl create project
command?
k
yes, you can. you will see the logs of propeller, admin, etc in the pod.
b
Here are the pods I have...
Copy code
$ kubectl get pods --all-namespaces
NAMESPACE                 NAME                                          READY   STATUS      RESTARTS   AGE
flyte                     flyte-backend-flyte-binary-786cf7cdf9-vpjt6   1/1     Running     0          141m
flytesnacks-development   ff4da8998ef014132b54-n0-0                     0/1     Completed   0          66s
flytesnacks-development   ff4da8998ef014132b54-n1-0                     0/1     Completed   0          66s
flytesnacks-development   ff4da8998ef014132b54-n2-0                     0/1     Completed   0          66s
kube-system               aws-node-c4x5t                                1/1     Running     0          5d1h
kube-system               aws-node-c6q29                                1/1     Running     0          5d1h
kube-system               aws-node-tw6n2                                1/1     Running     0          5d1h
kube-system               coredns-657694c6f4-7cbdr                      1/1     Running     0          5d1h
kube-system               coredns-657694c6f4-nxhpd                      1/1     Running     0          5d1h
kube-system               ebs-csi-controller-c869dc55b-bfqgm            6/6     Running     0          5d1h
kube-system               ebs-csi-controller-c869dc55b-q4nb7            6/6     Running     0          5d1h
kube-system               ebs-csi-node-ht6hr                            3/3     Running     0          5d1h
kube-system               ebs-csi-node-lzmkv                            3/3     Running     0          5d1h
kube-system               ebs-csi-node-x6pxr                            3/3     Running     0          5d1h
kube-system               efs-csi-controller-759b88757-cb97b            3/3     Running     0          23h
kube-system               efs-csi-controller-759b88757-dxv2x            3/3     Running     0          23h
kube-system               efs-csi-node-2w8hg                            3/3     Running     0          23h
kube-system               efs-csi-node-552xm                            3/3     Running     0          23h
kube-system               efs-csi-node-7fhlg                            3/3     Running     0          23h
kube-system               kube-proxy-7fdnj                              1/1     Running     0          5d1h
kube-system               kube-proxy-rfjck                              1/1     Running     0          5d1h
kube-system               kube-proxy-rx2mx                              1/1     Running     0          5d1h
If I look in
flyte-backend-flyte-binary
all I see are SQL statements from pod startup
Copy code
2023/04/20 05:05:35 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/finisher_api.go:509
[0.462ms] [rows:1] SELECT count(*) FROM pg_indexes WHERE tablename = 'artifacts' AND indexname = 'artifacts_dataset_uuid_idx' AND schemaname = CURRENT_SCHEMA()
{"metrics-prefix":"flyte:","certDir":"/var/run/flyte/certs","localCert":true,"listenPort":9443,"serviceName":"flyte-backend-flyte-binary-webhook","servicePort":443,"secretName":"flyte-backend-flyte-binary-webhook-secret","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"gcpSecretManager":{"sidecarImage":"<http://gcr.io/google.com/cloudsdktool/cloud-sdk:alpine|gcr.io/google.com/cloudsdktool/cloud-sdk:alpine>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2"}}
If I look in the task pods all I see is this...
Copy code
$ kubectl logs ff4da8998ef014132b54-n0-0 -n flytesnacks-development
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-vkny2wcs because the default path (/home/flytekit/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
tar: Removing leading `/' from member names
Is there somewhere else I should be looking, or is there a Log config I am meant to set?
k
The task failed due to OOM, you can check log in ff4da8998ef014132b54-n0-0. you can increase default memory
by change the configmap, there is a default task memory
b
The task itself succeeds and it has 500Mi memory assigned, I'm not that concerned about it specifically. I am looking for the propeller logs to support my other question which is find out why the cluster_resource_manager templates are not setting up pv, pvc for the flyte project and why the task is not mounting the pvc volume. I was hoping the propeller logs would show some info about the pv, pvc and mounts https://flyte-org.slack.com/archives/CP2HDHKE1/p1681911227391799
d
Hey @Blair Anson can you help me understand this a little better? Maybe break it down a bit, IIUC you're using the
cluster_resource_manager
in FlyteAdmin to create a PV - this is not being created? And then you're expecting Flyte to automatically detect the existence of the PV and mount it in k8s Pods? Or are you doing anything specific to mount? Can you link to any docs you've used for this?
From this conversation it sounds like the only problem is auto creating the PV and PVC now right? You have
PodTemplate
setup to automatically mount the volume for each task.
b
Thanks for looking into this. Yes I have a Pod template to mount the pvc volume. The issue is that I have to manually create the pv & pvc for each project. The cluster_resorce_manager templates are not creating the pv & pvc when a new project is created
d
@Blair Anson from the list of
Pods
you provided, the one you are looking at logs for is
flyte-backend-flyte-binary-786cf7cdf9-vpjt6
. This is all of Flyte compiled into a single binary - so specifically you will be searching for flyteadmin logs regarding the cluster resource manager.
b
Thanks, I was able to finally find the cluster resource logs in the EKS web console which are a big help.
112 Views