What is the recommended update procedure for flyte...
# flyte-deployment
r
What is the recommended update procedure for flyte-core? I'm trying to upgrade from 1.13.2 to 1.14.1 - it seems there are some database migrations involved. On our first try we let the new flyteadmin/run-migrations container run for ~35 minutes until we decided to roll back. Is it recommended to shutdown all other flyte related workloads while migrations are run? Is it just expected to take longer than ~30 minutes for these migrations to finish? There also seems to be no logging in the run-migrations container - at least not at log level 3.
f
It’s not expected to take this long, but depends. Not sure which migrations are being triggered and size of your database
h
Can you share the contents of the last row in the
migrations
table you have?
r
The last one by date prefix is
2024-06-06-execution-tags
but there are others - see screenshot
a
Also if you set logging level to 5 (debug) do you see anything useful?
r
I did not try setting the logging level to 5 yet. Also I just found out that another team is running pretty heavy select queries on the flyte db to get some statistics about their workflow. I've asked them to stop since 1. It's not a great idea 2. It might interfere with the database migrations. So I'll try again once those queries are no longer running...
f
@ripe-smartphone-56353 in union we have cost observability built in and also other stats at the workflow level
r
Just to cycle back here. The upgrade did succeed after letting the migrations run for ~70 minutes. I did run the upgrade with log level 5 but there was no output so it was a bit đŸ˜° Seems I just wasn't patient enough the first time around.
f
How many executions do you have
r
Count of the
executions
table is ~3.2 Milliion
f
That’s not a lot
Good to know