https://flyte.org
Join Slack
Hi team.. question about Flyte backend. Does the propeller inject imagePullSecrets into the task for...
# flyte-support
l
Hi team.. question about Flyte backend. Does the propeller inject imagePullSecrets into the task for it to actually pull the workflows docker image for the pod?
t
image pull secrets are usually controlled by the relevant service account to the best of my knowledge.
using the flyte templating functionality (the cluster resource controller) you can customize these service accounts
but otherwise as far as i know, propeller is not involved with explicitly setting image pull secrets (pod templates notwithstanding)
l
We are seeing a weird case where some workflows are getting the appropriate SA (and image pull secrets)… and some are not.. and the 2nd group is failing. I’m not super familiar with the core setup, hence trying to figure out what the issue may be
t
are they in the same namespace?
do you have instances of the same workflow in the same project/domain exhibiting different behavior across executions? or is it different workflows?
l
k8s namespace: yes Flyte namespace: no
At this point different workflows
I can try the same one
t
are there matchable resource overrides at any layer (project, project&domain, project&domain&workflow) for these workflows?
and are there any podtemplates in the cluster? (either in the flyte namespace or in the namespace the workflow is running in)
l
The one that’s running successfully is using a pod template etc The one that’s failing is using the default pod template that we’ve defined
Same workflow both namespaces.. fails the same way (unable to pull docker image due to missing credentials) Same version.. different workflow.. seems to have the credentials. Not sure if this matters context wise but yesterday we did upgrade flyte to 1.5.1
t
is the service account getting assigned correctly?
l
The task that appears to not have a problem is using a runtime pod template for a container task.. and appears to have the service account (and credentials) correctly
Meanwhile everything else fails with something like this
Copy code
Last Error: USER::containers with unready status: [al28hgkzj65k7zhmwgdq-n0-0]|Back-off pulling image "<http://mydocker.com/flyte/automation:cb62f19b6798ce4be81ffae526112031083f919f|mydocker.com/flyte/automation:cb62f19b6798ce4be81ffae526112031083f919f>"
t
if you run a hello world workflow/task using the custom image, it will fail to have the correct service account and hence fail to pull the custom image in all namespaces and - is this true?
l
Apparently the docker pull requests are being made sans secrets
yes.. looks like namespace isn’t the problem here..
t
what service account did al28hgkzj65k7zhmwgdq-n0-0 run with?
flyte 1.5.1 is about a year old already. what version did you upgrade from?
l
something even older (1.3) 😬
t
understood. will have to look through the changelogs, definitely don’t remember changes that far back
the al28hgkzj65k7zhmwgdq run though, did that run with the correct SA?
l
ok so the failed one seems to have used SA “default” the other used the correct SA “flytetask”
t
my first inclination is something to do with pod templates. i feel like at the time a lot of work around that was being done.
and default is not the correct SA i assume.
are you setting the k8sServiceAccount configuration in Admin?
l
yes
Copy code
flyteadmin:
      interruptible: true
      k8sServiceAccount: flytetask
I may have found something.. let me try
No luck.. somehow even though the the service account is set, the actual pod yaml doesn’t have any of it.
Resolved.. some side effect of the upgrade. Somehow a bunch of pods were not getting the right SA.. so we’ve now added it into the default pod template
🙏🏽 1
🙏 1
13 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by