https://flyte.org
Join Slack
How do you change imagePullPolicy for existing deployments? By default, its ifnotpresent, but I woul...
# flyte-support
n
How do you change imagePullPolicy for existing deployments? By default, its ifnotpresent, but I would like to change it to always for all of my potential docker images/containers. Do I have to use https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages?
f
use - PodTemplate to adjust the policy
g
Here is an example
n
Copy code
kubectl apply -f - <<EOF
apiVersion: v1
kind: PodTemplate
metadata:
  name: flyte-template
  namespace: flyte
template:
  spec:
    containers:
        - name: default
          image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
          imagePullPolicy: Always
EOF
something like this is on the per container level right? so I have to add one for each possible container I want to use?
g
yes
n
I created a pod template named 
flyte-template
under the namespace 
flyte
. And then added it to the helm chart. 
kubectl get ConfigMap -n flyte flyte-propeller-config -o yaml
returns
Copy code
k8s.yaml: |
  plugins:
   k8s:
    default-cpus: 100m
    default-env-vars: []
    default-memory: 100Mi
    default-pod-template-name: flyte-template
but new task pods still have IfNotPresent when I view in 
kubectl edit pod aflvhfnpb4hjb4rtxnnz-fzmlagaq-0 -n flytetester-development -o yaml
verified that the image in the task yaml is one that was specified as Always in the pod template
Oh I don't think the pod template is being used at all because I added a new random label and I don't see it in new pods
f
Cc @hallowed-mouse-14616
h
Hi @numerous-hamburger-7178, a few questions: (1) what version of FlytePropeller are you running? (2) What namespace if FlytePropeller running in?
n
is copilot required?
h
it should not be.
n
flytepropeller is running in flyte namespace. how do I check the version? this is what I see in the pod log
Copy code
"App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2022-11-02 18:05:16.617334666 +0000 UTC m=+0.030357149]"
h
can you see what image is running in the Pod?
n
Copy code
Container image "<http://cr.flyte.org/flyteorg/flytepropeller-release:v1.2.0|cr.flyte.org/flyteorg/flytepropeller-release:v1.2.0>" already present on machine
h
and what time did the container start?
n
8 minutes ago
Copy code
<http://helm.sh/chart|helm.sh/chart>	flyte-core-v1.2.0
h
oh perfect 😆 so it's not a stale version of propeller!
Can you try to create that same PodTemplate (with the label test you were trying) in the namespace that the Pod will be created in? So in your test it looks like 
flytetester-development
.
n
did that too. no difference
👍 1
h
OK, I'm diving into this and having trouble figuring out what could be going wrong. Just to reclarify, the Pod has been restarted since the configuration change right? This isn't a configuration update the FlytePropeller can process while running.
n
the flytepropeller pods have been restarted since I updated the helm chart. and I am kicking off new pods using flyte ui
👍 1
@hallowed-mouse-14616 jumped on a call and helped! turns out I wasn't setting the imagePullPolicy field under default/primary container. I was just adding it to new containers based on the images I was using.
thanks a lot for the help everyone 🙏
❤️ 2
181 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by