Hi we re running a bunch of large jobs each with thousands o

Question

Hi we re running a bunch of large jobs each with thousands of tasks and noticed our Postgres database was at 100 CPU for many hours DB only has 4 vCPU and 15GB of memory at the moment Saw many of these in the datacatalog log as well ``` textPayload 2023 03 07 15 38 00 u001b 32m go src github com flyteorg datacatalog pkg repositories gormimpl artifact go 64 u001b 33mSLOW SQL gt = 200ms ``` Do you have a recommended size for postgres for large production workloads I m also going through <https docs flyte org en latest deployment configuration performance html optimizing performance|this doc> to make sure we re following best practices

Nicholas LoFaso · Answer

Also seeing many of ```2023 03 07 16 14 58 742 UTC 1644473 4 1 db=datacatalog user=flyteadmin ERROR duplicate key value violates unique constraint tags pkey ``` And ```2023 03 07 16 14 58 874 UTC 1644473 7 1 db=datacatalog user=flyteadmin ERROR duplicate key value violates unique constraint datasets pkey ``` In our postgres logs Any idea why those are showing up

Ketan (kumare3) · Answer

< Nicholas LoFaso> is this in datacatalog we have not seen these issues

Ketan (kumare3) · Answer

ya for larger workloads DB can run how but 100 cpu for datacatalog seems odd

Ketan (kumare3) · Answer

can you share more

Ketan (kumare3) · Answer

happy to hop on a call to help

Nicholas LoFaso · Answer

Yes that was a datacatalog error The 100 CPU was for our managed postgres instance not the data manager itself

Nicholas LoFaso · Answer

I m on the east coast available for a call any time tomorrow

Ketan (kumare3) · Answer

are you using same postgres for both datacatalog and admin

Nicholas LoFaso · Answer

yes same for both

Nicholas LoFaso · Answer

We significantly increased the size of postgres and that resolved the current bottleneck I will be digging into our Flyte performance over the next couple of weeks so will likely have additional questions but for now I plan to setup prometheus to gather the data plane metrics

Ketan (kumare3) · Answer

yup

Ketan (kumare3) · Answer

but it still seems odd that postgres was hammered

Ketan (kumare3) · Answer

we would love to understand your workflow pattern

Nicholas LoFaso · Answer

We would love to share and improve our pattern configuration Maybe we can setup a call late next week after I ve had a chance to gather some metrics