Hi, we’re running a bunch of large jobs each with ...
# flyte-supportr
rough-rose-81585
03/09/2023, 7:12 PMHi, we’re running a bunch of large jobs each with thousands of tasks, and noticed our Postgres database was at 100% CPU for many hours. DB only has 4 vCPU and 15GB of memory at the moment.
Saw many of these in the datacatalog log as well
Copy code
"textPayload": "2023/03/07 15:38:00 \u001b[32m/go/src/github.com/flyteorg/datacatalog/pkg/repositories/gormimpl/artifact.go:64 \u001b[33mSLOW SQL >= 200ms",rough-rose-81585
03/09/2023, 9:04 PMAlso seeing many of
Copy code
2023-03-07 16:14:58.742 UTC [1644473]: [4-1] db=datacatalog,user=flyteadmin ERROR: duplicate key value violates unique constraint "tags_pkey" Copy code
2023-03-07 16:14:58.874 UTC [1644473]: [7-1] db=datacatalog,user=flyteadmin ERROR: duplicate key value violates unique constraint "datasets_pkey"f
freezing-airport-6809
03/10/2023, 12:59 AM
@rough-rose-81585 is this in datacatalog.
we have not seen these issues
freezing-airport-6809
03/10/2023, 12:59 AM
ya for larger workloads DB can run how - but 100% cpu for datacatalog seems odd
freezing-airport-6809
03/10/2023, 12:59 AM
can you share more
freezing-airport-6809
03/10/2023, 12:59 AM
happy to hop on a call to help
r
rough-rose-81585
03/10/2023, 2:08 AMYes that was a datacatalog error. The 100% CPU was for our managed postgres instance not the data manager itself
rough-rose-81585
03/10/2023, 2:08 AMI’m on the east coast available for a call any time tomorrow
f
freezing-airport-6809
03/10/2023, 5:57 AM
are you using same postgres for both datacatalog and admin?
r
rough-rose-81585
03/10/2023, 3:54 PMyes same for both
rough-rose-81585
03/10/2023, 7:26 PMWe significantly increased the size of postgres and that resolved the current bottleneck. I will be digging into our Flyte performance over the next couple of weeks so will likely have additional questions but for now I plan to setup prometheus to gather the data plane metrics
f
freezing-airport-6809
03/10/2023, 9:26 PM
yup
freezing-airport-6809
03/10/2023, 9:26 PM
but it still seems odd that postgres was hammered
freezing-airport-6809
03/10/2023, 9:26 PM
we would love to understand your workflow pattern
r
rough-rose-81585
03/10/2023, 9:45 PMWe would love to share and improve our pattern / configuration. Maybe we can setup a call late next week after I’ve had a chance to gather some metrics?
👍 1
154 Views