hi there, i’m trying to pull an image from a private registry. i modified the default service account to look like this:

kubectl edit serviceaccount/default -n flyte

apiVersion: v1
imagePullSecrets:
- name: ghcr
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-08-18T20:42:23Z"
  name: default
  namespace: flyte
  resourceVersion: "13612623"
  uid: 5bc39079-a6a5-4455-ae27-31eaed46c368
secrets:
- name: default-token-mfw8r

then created a secret resource to look like this:

kubectl edit secret/ghcr -n flyte

apiVersion: v1
data:
  .dockerconfigjson: <REDACTED>
kind: Secret
metadata:
  annotations:
    <http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: |
      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"ghcr","namespace":"flyte"},"stringData":{".dockerconfigjson":"{\"auths\":{\"<http://ghcr.io|ghcr.io>\":{\"username\":\"<REDACTED>\",\"password\":\"<REDACTED>\",\"email\":\"<REDACTED>\",\"auth\":\"<REDACTED>\"}}}"},"type":"<http://kubernetes.io/dockerconfigjson|kubernetes.io/dockerconfigjson>"}
  creationTimestamp: "2022-08-25T21:02:59Z"
  name: ghcr
  namespace: flyte
  resourceVersion: "13628687"
  uid: 24dc477e-71cc-42eb-a0be-1bc61af20f5c
type: <http://kubernetes.io/dockerconfigjson|kubernetes.io/dockerconfigjson>

i used the

config.json

locally and was able to push and pull images to and from the private registry. but when executing a workflow, i get an image pull error in the UI (below) and when i describe the pod, i get this:

Normal   Scheduled  18m                   default-scheduler  Successfully assigned flytesnacks-development/ajjf7v4swhdz7jxzvhwc-n0-0 to ip-10-15-147-196.ec2.internal
  Normal   Pulling    17m (x4 over 18m)     kubelet            Pulling image "<http://ghcr.io/predictap/symphony_hall:v0.0.14|ghcr.io/predictap/symphony_hall:v0.0.14>"
  Warning  Failed     17m (x4 over 18m)     kubelet            Failed to pull image "<http://ghcr.io/predictap/symphony_hall:v0.0.14|ghcr.io/predictap/symphony_hall:v0.0.14>": rpc error: code = Unknown desc = Error response from daemon: Head "<https://ghcr.io/v2/predictap/symphony_hall/manifests/v0.0.14>": unauthorized
  Warning  Failed     17m (x4 over 18m)     kubelet            Error: ErrImagePull
  Warning  Failed     17m (x6 over 18m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff    3m26s (x65 over 18m)  kubelet            Back-off pulling image "<http://ghcr.io/predictap/symphony_hall:v0.0.14|ghcr.io/predictap/symphony_hall:v0.0.14>"

is there a good way to troubleshoot this? it seems like the docker config may not be present on the node that’s trying to pull the image (maybe one of the ec2 nodes associated with the EKS cluster?)

you should patch the

default

service account under the

PROJECTNAME-development

namespace instead of “Flyte”. https://github.com/flyteorg/flyte/discussions/1107#discussioncomment-842776

ah nice. ok, and probably should create the secret in that namespace too.

yes, correct. 👍