Hi, when using flyte-binary how can I view the flyte-propeller logs to see any errors with workflow execution? And what about project creation, are there logs on the server showing changes done as the result of a

flytectl create project

command?

yes, you can. you will see the logs of propeller, admin, etc in the pod.

Here are the pods I have...

$ kubectl get pods --all-namespaces
NAMESPACE                 NAME                                          READY   STATUS      RESTARTS   AGE
flyte                     flyte-backend-flyte-binary-786cf7cdf9-vpjt6   1/1     Running     0          141m
flytesnacks-development   ff4da8998ef014132b54-n0-0                     0/1     Completed   0          66s
flytesnacks-development   ff4da8998ef014132b54-n1-0                     0/1     Completed   0          66s
flytesnacks-development   ff4da8998ef014132b54-n2-0                     0/1     Completed   0          66s
kube-system               aws-node-c4x5t                                1/1     Running     0          5d1h
kube-system               aws-node-c6q29                                1/1     Running     0          5d1h
kube-system               aws-node-tw6n2                                1/1     Running     0          5d1h
kube-system               coredns-657694c6f4-7cbdr                      1/1     Running     0          5d1h
kube-system               coredns-657694c6f4-nxhpd                      1/1     Running     0          5d1h
kube-system               ebs-csi-controller-c869dc55b-bfqgm            6/6     Running     0          5d1h
kube-system               ebs-csi-controller-c869dc55b-q4nb7            6/6     Running     0          5d1h
kube-system               ebs-csi-node-ht6hr                            3/3     Running     0          5d1h
kube-system               ebs-csi-node-lzmkv                            3/3     Running     0          5d1h
kube-system               ebs-csi-node-x6pxr                            3/3     Running     0          5d1h
kube-system               efs-csi-controller-759b88757-cb97b            3/3     Running     0          23h
kube-system               efs-csi-controller-759b88757-dxv2x            3/3     Running     0          23h
kube-system               efs-csi-node-2w8hg                            3/3     Running     0          23h
kube-system               efs-csi-node-552xm                            3/3     Running     0          23h
kube-system               efs-csi-node-7fhlg                            3/3     Running     0          23h
kube-system               kube-proxy-7fdnj                              1/1     Running     0          5d1h
kube-system               kube-proxy-rfjck                              1/1     Running     0          5d1h
kube-system               kube-proxy-rx2mx                              1/1     Running     0          5d1h

If I look in

flyte-backend-flyte-binary

all I see are SQL statements from pod startup

2023/04/20 05:05:35 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/finisher_api.go:509
[0.462ms] [rows:1] SELECT count(*) FROM pg_indexes WHERE tablename = 'artifacts' AND indexname = 'artifacts_dataset_uuid_idx' AND schemaname = CURRENT_SCHEMA()
{"metrics-prefix":"flyte:","certDir":"/var/run/flyte/certs","localCert":true,"listenPort":9443,"serviceName":"flyte-backend-flyte-binary-webhook","servicePort":443,"secretName":"flyte-backend-flyte-binary-webhook-secret","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"gcpSecretManager":{"sidecarImage":"<http://gcr.io/google.com/cloudsdktool/cloud-sdk:alpine|gcr.io/google.com/cloudsdktool/cloud-sdk:alpine>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2"}}

If I look in the task pods all I see is this...

$ kubectl logs ff4da8998ef014132b54-n0-0 -n flytesnacks-development
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-vkny2wcs because the default path (/home/flytekit/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
tar: Removing leading `/' from member names

Is there somewhere else I should be looking, or is there a Log config I am meant to set?

The task failed due to OOM, you can check log in ff4da8998ef014132b54-n0-0. you can increase default memory

The task itself succeeds and it has 500Mi memory assigned, I'm not that concerned about it specifically. I am looking for the propeller logs to support my other question which is find out why the cluster_resource_manager templates are not setting up pv, pvc for the flyte project and why the task is not mounting the pvc volume. I was hoping the propeller logs would show some info about the pv, pvc and mounts https://flyte-org.slack.com/archives/CP2HDHKE1/p1681911227391799

Hey @bitter-hydrogen-618 can you help me understand this a little better? Maybe break it down a bit, IIUC you're using the

cluster_resource_manager

in FlyteAdmin to create a PV - this is not being created? And then you're expecting Flyte to automatically detect the existence of the PV and mount it in k8s Pods? Or are you doing anything specific to mount? Can you link to any docs you've used for this?

Thanks for looking into this. Yes I have a Pod template to mount the pvc volume. The issue is that I have to manually create the pv & pvc for each project. The cluster_resorce_manager templates are not creating the pv & pvc when a new project is created

@bitter-hydrogen-618 from the list of

Pods

you provided, the one you are looking at logs for is

flyte-backend-flyte-binary-786cf7cdf9-vpjt6

. This is all of Flyte compiled into a single binary - so specifically you will be searching for flyteadmin logs regarding the cluster resource manager.

Thanks, I was able to finally find the cluster resource logs in the EKS web console which are a big help.