Does anybody ever encountered datacatalog pod hang...
# flyte-supportb
brief-leather-11142
11/14/2022, 1:52 PMDoes anybody ever encountered datacatalog pod hanging in the init container stage? It seems my pod stuck in the waiting status forever. Yet once I deleted the pod init container, the pod is up.
Besides I’m wondering what does the datacatalog migrate actually do, would it be ok to skip this step if no db changes occurs?
h
hallowed-mouse-14616
11/14/2022, 3:21 PMI think you understand correctly. The datacatalog migrate step updates the DB schema if necessary, adding / removing fields and / or tables. Can you provide a little more information on what is stuck? Can you get logs from the init container? dump the Pod yaml from k8s?
f
freezing-airport-6809
11/14/2022, 3:44 PM
Hmm stuck usually would mean unable to reach the database
freezing-airport-6809
11/14/2022, 3:44 PM
Logs would really help
b
brief-leather-11142
11/14/2022, 3:56 PMI didn’t find any log anywhere as the container is not even up. K8s event says it’s waiting for image pulling.
brief-leather-11142
11/14/2022, 3:57 PMI am pretty sure the registry is set up properly as once I removed the datacatalog migrate step, the pod is back to normal
brief-leather-11142
11/14/2022, 4:03 PMAnd the logs are the following:
Copy code
time="2022-11-14T13:11:41Z" level=info msg="Using config file: [/etc/datacatalog/config/db.yaml /etc/datacatalog/config/logger.yaml /etc/datacatalog/config/server.yaml /etc/datacatalog/config/storage.yaml]"
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [database] updated. No update handler registered.","ts":"2022-11-14T13:11:41Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [datacatalog] updated. No update handler registered.","ts":"2022-11-14T13:11:41Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [application] updated. No update handler registered.","ts":"2022-11-14T13:11:41Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-11-14T13:11:41Z"}
{"json":{"src":"service.go:160"},"level":"info","msg":"Serving DataCatalog http on port :8080","ts":"2022-11-14T13:11:41Z"}
{"json":{"src":"stow_store.go:396"},"level":"warning","msg":"stow configuration section missing, defaulting to legacy s3/minio connection config","ts":"2022-11-14T13:11:41Z"}
{"json":{"app_name":"datacatalog","src":"service.go:92"},"level":"info","msg":"Created data storage.","ts":"2022-11-14T13:11:46Z"}
{"json":{"app_name":"datacatalog","src":"service.go:103"},"level":"info","msg":"Created DB connection.","ts":"2022-11-14T13:11:46Z"}
{"json":{"src":"server.go:96"},"level":"info","msg":"Starting profiling server on port [10254]","ts":"2022-11-14T13:11:46Z"}
{"json":{"src":"service.go:132"},"level":"info","msg":"Serving DataCatalog Insecure on port :8089","ts":"2022-11-14T13:11:46Z"}h
hallowed-mouse-14616
11/14/2022, 4:08 PMwhat is the image for the init container? it doesn't match the image for regular container?
b
brief-leather-11142
11/14/2022, 4:09 PMit’s the same, I deployed it from the flyte helm chart.
h
hallowed-mouse-14616
11/14/2022, 4:10 PMOh OK, then it doesn't make sense that k8s would be waiting for image pulling.
b
brief-leather-11142
11/14/2022, 4:11 PMya. I guess the image was pulled yet it stuck in migrating
brief-leather-11142
11/14/2022, 4:12 PMDoes the migration have anything to do with the object store?
h
hallowed-mouse-14616
11/14/2022, 4:14 PMit should not. it just creates the database and correct schema to store the information datacatalog requires. This is the first time you have datacatalog running then right? Like you're setting up Flyte?
hallowed-mouse-14616
11/14/2022, 4:15 PMCan you confirm that the postgres instance that is defined in the datacatalog configuration is running and accessible?
b
brief-leather-11142
11/14/2022, 4:16 PMya,first time. I tried to re-deploy the datacatalog couple of times. Neither of those worked.
brief-leather-11142
11/14/2022, 4:16 PMI am using an RDS instance, it looks pretty healthy
brief-leather-11142
11/14/2022, 4:18 PMFlyteadmin was pretty ok with the db as I was able to run various flytectl get commands to check the metadata at the same time.
h
hallowed-mouse-14616
11/14/2022, 4:23 PMOK, if that's the case then I suspect there is some issue with the datacatalog DB connection code. I'm looking through it a little bit here.
hallowed-mouse-14616
11/14/2022, 4:24 PMDo you mind dumping the db configuration from both flyteadmin and datacatalog? please replcace any sensitive fields (hostnames, etc) with random placeholders.
b
brief-leather-11142
11/14/2022, 4:25 PMSure. I am pretty sure I used the same set up for those two though as they are configured in the same place in the helm charts.
brief-leather-11142
11/14/2022, 4:25 PMIt’s in the cm right ?
h
hallowed-mouse-14616
11/14/2022, 4:26 PMshould be. i'll be honest, i'm not intimately familiar with the helm charts. maybe i should take a look there quick 😄
b
brief-leather-11142
11/14/2022, 4:27 PMdb.yaml: |
database:
dbname: datacatalog
host: pgm-*
passwordPath: /etc/db/pass.txt
port: 5432
username: flyteadmin
db.yaml: |
database:
dbname: flyteadmin
host: pgm-*
passwordPath: /etc/db/pass.txt
port: 5432
username: flyteadmin
brief-leather-11142
11/14/2022, 4:27 PMit’s basically the same
👍 1
h
hallowed-mouse-14616
11/14/2022, 7:23 PM156 Views