Hi, we’re running a bunch of large jobs each with ...
# ask-the-communityn
Nicholas LoFaso
03/09/2023, 7:12 PMHi, we’re running a bunch of large jobs each with thousands of tasks, and noticed our Postgres database was at 100% CPU for many hours. DB only has 4 vCPU and 15GB of memory at the moment.
Saw many of these in the datacatalog log as well
Copy code
"textPayload": "2023/03/07 15:38:00 \u001b[32m/go/src/github.com/flyteorg/datacatalog/pkg/repositories/gormimpl/artifact.go:64 \u001b[33mSLOW SQL >= 200ms",Nicholas LoFaso
03/09/2023, 9:04 PMAlso seeing many of
Copy code
2023-03-07 16:14:58.742 UTC [1644473]: [4-1] db=datacatalog,user=flyteadmin ERROR: duplicate key value violates unique constraint "tags_pkey" Copy code
2023-03-07 16:14:58.874 UTC [1644473]: [7-1] db=datacatalog,user=flyteadmin ERROR: duplicate key value violates unique constraint "datasets_pkey"k
Ketan (kumare3)
03/10/2023, 12:59 AM
@Nicholas LoFaso is this in datacatalog.
we have not seen these issues
Ketan (kumare3)
03/10/2023, 12:59 AM
ya for larger workloads DB can run how - but 100% cpu for datacatalog seems odd
Ketan (kumare3)
03/10/2023, 12:59 AM
can you share more
Ketan (kumare3)
03/10/2023, 12:59 AM
happy to hop on a call to help
n
Nicholas LoFaso
03/10/2023, 2:08 AMYes that was a datacatalog error. The 100% CPU was for our managed postgres instance not the data manager itself
Nicholas LoFaso
03/10/2023, 2:08 AMI’m on the east coast available for a call any time tomorrow
k
Ketan (kumare3)
03/10/2023, 5:57 AM
are you using same postgres for both datacatalog and admin?
n
Nicholas LoFaso
03/10/2023, 3:54 PMyes same for both
Nicholas LoFaso
03/10/2023, 7:26 PMWe significantly increased the size of postgres and that resolved the current bottleneck. I will be digging into our Flyte performance over the next couple of weeks so will likely have additional questions but for now I plan to setup prometheus to gather the data plane metrics
k
Ketan (kumare3)
03/10/2023, 9:26 PM
yup
Ketan (kumare3)
03/10/2023, 9:26 PM
but it still seems odd that postgres was hammered
Ketan (kumare3)
03/10/2023, 9:26 PM
we would love to understand your workflow pattern
n
Nicholas LoFaso
03/10/2023, 9:45 PMWe would love to share and improve our pattern / configuration. Maybe we can setup a call late next week after I’ve had a chance to gather some metrics?
152 Views