Has anyone been able to get flyte running in a system with istio? I'm running into issues with init containers and wondering if there are any known workarounds

Hey Noah. What issues and what init containers? There have been issues in the past with Istio and the

flytescheduler-check

init container

We have issues with all of the init containers, so this would be • flyteadmin: run-migrations • datacatalog: run-migrations • flytescheduler: flytescheduler-check • flytepropeller: generate-secrets There's configuration to disable the flytescheduler-check, but not any of these other containers. I'm also not sure of the implications of disabling these containers, it seems like many of them are necessary. I'm currently exploring moving them to helm pre install hooks, but looking for better alternatives

cc @handsome-sandwich-69169 / @stocky-notebook-88311 and others who I think are Istio users. Also, I don't think disabling all those init containers would be safe 🙂

We went flyte binary route and we didn’t see this type of issues. However we found an issue with the istio sidecar on workflows pods sticking around but we found one option flag on flyte to allow it to remove them -> (edit:

inject-finalizer

)

Thanks @handsome-sandwich-69169! Given flyte-binary runs most of the services in a single container, that makes sense... though I wonder in your installation if you've disabled the init container that performs the db check? https://github.com/flyteorg/flyte/blob/master/charts/flyte-binary/templates/deployment.yaml#L63-L97

No really, it did run

Ah, interesting. And out of curiosity, are you on k8s 1.29 and using the new sidecar container support for Istio?

No yet

I wonder if you're using the uid

1337

workaround by any chance?

I didn't know about it, but looks like our istio pods have the

runasUser: 1337

Ah yup, that would explain it :) Someone ran into the istio problem and fixed it in your environment