What is the recommended update procedure for flyte-core? I'm trying to upgrade from 1.13.2 to 1.14.1 - it seems there are some database migrations involved. On our first try we let the new flyteadmin/run-migrations container run for ~35 minutes until we decided to roll back. Is it recommended to shutdown all other flyte related workloads while migrations are run? Is it just expected to take longer than ~30 minutes for these migrations to finish? There also seems to be no logging in the run-migrations container - at least not at log level 3.

It’s not expected to take this long, but depends. Not sure which migrations are being triggered and size of your database

Can you share the contents of the last row in the

migrations

table you have?

The last one by date prefix is

2024-06-06-execution-tags

but there are others - see screenshot

Also if you set logging level to 5 (debug) do you see anything useful?

I did not try setting the logging level to 5 yet. Also I just found out that another team is running pretty heavy select queries on the flyte db to get some statistics about their workflow. I've asked them to stop since 1. It's not a great idea 2. It might interfere with the database migrations. So I'll try again once those queries are no longer running...

@ripe-smartphone-56353 in union we have cost observability built in and also other stats at the workflow level

Just to cycle back here. The upgrade did succeed after letting the migrations run for ~70 minutes. I did run the upgrade with log level 5 but there was no output so it was a bit 😰 Seems I just wasn't patient enough the first time around.

How many executions do you have

Count of the

executions

table is ~3.2 Milliion

That’s not a lot