@average-finland-92144 I'm planning to start soon on the work to upstream changes to support Istio in the Helm chart. A couple more questions about this: • Init containers (like for initializing secrets, performing db migrations, etc) have to become k8s jobs - there really isn't a great way around this due to some of the problems Istio presents around accessing other services over the network • Those jobs have to be setup to run as Helm hooks to sequence operations correctly (flyteadmin will crashloop until secrets are setup on first run, for instance - which can be nondeterminstic without using a hook) • I see a bunch of other in-flight charts PRs against the project right now. Might be better to get those in first before I perform any more surgery Thoughts? Thanks!

Can we not add istio?

Just to clarify... I'm not adding Istio to the charts. I'm actually not a fan of Istio, so you're preaching to the choir 😉

I'm concerned too about the changes from initContainers to K8s jobs and then Helm hooks. How configurable would be that? I mean, like an opt-in to avoid breaking current behavior

The intent would be to not break current behavior on upgrades -- are there specific init containers you're concerned about?

I had exactly

db-migrations

in mind. AFAICT there are some prefixes in place to prevent clashes at the

gorm

level but not sure if it has to do with the situation of multiple flyteadmin instances. Maybe @high-accountant-32689 knows better 🙂

Yeah, running migrations in an initContainer when you have multiple replicas of a k8s deployment can be... problematic (especially if the migrations aren't versioned, don't run inside transactions, etc). It's usually best to decouple them and execute in a job separate from the service

Thankfully the migrations are versioned and run in transactions, we use https://github.com/go-gormigrate/gormigrate to accomplish this. Unfortunately, we do not have a mechanism in place to prevent multiple migrations from competing. I'd be down to helping decouple this (it could be as simple as sharing a redis instance to hold a distributed lock during migrations).