https://flyte.org logo
Title
l

Laura Lin

11/01/2022, 6:51 PM
How do you change imagePullPolicy for existing deployments? By default, its ifnotpresent, but I would like to change it to always for all of my potential docker images/containers. Do I have to use https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages?
k

Ketan (kumare3)

11/01/2022, 8:29 PM
use - PodTemplate to adjust the policy
k

Kevin Su

11/01/2022, 8:49 PM
Here is an example
l

Laura Lin

11/01/2022, 9:01 PM
kubectl apply -f - <<EOF
apiVersion: v1
kind: PodTemplate
metadata:
  name: flyte-template
  namespace: flyte
template:
  spec:
    containers:
        - name: default
          image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
          imagePullPolicy: Always
EOF
something like this is on the per container level right? so I have to add one for each possible container I want to use?
k

Kevin Su

11/01/2022, 9:05 PM
yes
l

Laura Lin

11/01/2022, 11:16 PM
I created a pod template named
flyte-template
under the namespace
flyte
. And then added it to the helm chart.
kubectl get ConfigMap -n flyte flyte-propeller-config -o yaml
returns
k8s.yaml: |
  plugins:
   k8s:
    default-cpus: 100m
    default-env-vars: []
    default-memory: 100Mi
    default-pod-template-name: flyte-template
but new task pods still have IfNotPresent when I view in
kubectl edit pod aflvhfnpb4hjb4rtxnnz-fzmlagaq-0 -n flytetester-development -o yaml
verified that the image in the task yaml is one that was specified as Always in the pod template
Oh I don't think the pod template is being used at all because I added a new random label and I don't see it in new pods
k

Ketan (kumare3)

11/02/2022, 5:45 PM
Cc @Dan Rammer (hamersaw)
d

Dan Rammer (hamersaw)

11/02/2022, 6:04 PM
Hi @Laura Lin, a few questions: (1) what version of FlytePropeller are you running? (2) What namespace if FlytePropeller running in?
l

Laura Lin

11/02/2022, 6:04 PM
is copilot required?
d

Dan Rammer (hamersaw)

11/02/2022, 6:05 PM
it should not be.
l

Laura Lin

11/02/2022, 6:06 PM
flytepropeller is running in flyte namespace. how do I check the version? this is what I see in the pod log
"App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2022-11-02 18:05:16.617334666 +0000 UTC m=+0.030357149]"
d

Dan Rammer (hamersaw)

11/02/2022, 6:13 PM
can you see what image is running in the Pod?
l

Laura Lin

11/02/2022, 6:13 PM
Container image "<http://cr.flyte.org/flyteorg/flytepropeller-release:v1.2.0|cr.flyte.org/flyteorg/flytepropeller-release:v1.2.0>" already present on machine
d

Dan Rammer (hamersaw)

11/02/2022, 6:14 PM
and what time did the container start?
l

Laura Lin

11/02/2022, 6:14 PM
8 minutes ago
<http://helm.sh/chart|helm.sh/chart>	flyte-core-v1.2.0
d

Dan Rammer (hamersaw)

11/02/2022, 6:14 PM
oh perfect 😆 so it's not a stale version of propeller!
Can you try to create that same PodTemplate (with the label test you were trying) in the namespace that the Pod will be created in? So in your test it looks like
flytetester-development
.
l

Laura Lin

11/02/2022, 6:16 PM
did that too. no difference
d

Dan Rammer (hamersaw)

11/02/2022, 6:52 PM
OK, I'm diving into this and having trouble figuring out what could be going wrong. Just to reclarify, the Pod has been restarted since the configuration change right? This isn't a configuration update the FlytePropeller can process while running.
l

Laura Lin

11/02/2022, 6:54 PM
the flytepropeller pods have been restarted since I updated the helm chart. and I am kicking off new pods using flyte ui
@Dan Rammer (hamersaw) jumped on a call and helped! turns out I wasn't setting the imagePullPolicy field under default/primary container. I was just adding it to new containers based on the images I was using.
thanks a lot for the help everyone 🙏