https://flyte.org logo
Title
z

Zachary Kimble

09/16/2022, 5:22 PM
Hello, I'm trying to run a demo workflow with a custom image in local sandbox, but am having authentication issues puling the image from Azure container registry. I initialized the sandbox with
flytectl sandbox start
I created a secrets file
acr-secrets-flytesnacks-development.yaml
apiVersion: v1
data:
  .dockerconfigjson: ***Base64 encoded json***
kind: Secret
metadata:
  name: acr-pull-credentials
  namespace: flyte-development
type: <http://kubernetes.io/dockerconfigjson|kubernetes.io/dockerconfigjson>
I applied the secret in the project-domain namespace with
kubectl -n flytesnacks-development apply -f secrets/acr-secrets-flytesnacks-development.yaml
I patched the default service account within the project-domain namespace :
kubectl -n flytesnacks-development patch serviceaccount default -p '{"imagePullSecrets": [{"name": "acr-pull-credentials"}]}'
I run my workflow with
pyflyte run --remote --image ***.<http://azurecr.io/databricks_workflow:latest|azurecr.io/databricks_workflow:latest> databricks_wf.py databricks_workflow --sql 'select...'
The pods get stuck in pending and describe shows the following issue:
Failed to pull image "***.<http://azurecr.io/databricks_workflow:latest|azurecr.io/databricks_workflow:latest>": rpc error: code = Unknown desc = Error response from daemon: Head "https://*****.<http://azurecr.io/v2/databricks_workflow/manifests/latest|azurecr.io/v2/databricks_workflow/manifests/latest>": unauthorized: authentication required, visit <https://aka.ms/acr/authorization> for more information.
Placing the decoded secret in .docker/config.json allows
docker pull
to work locally. Thanks for a any suggestions!
y

Yee

09/16/2022, 8:56 PM
can you check the pod?
get the
-o yaml
for the pod and grep for pull secrets?
z

Zachary Kimble

09/16/2022, 9:00 PM
I don't understand what command you mean by check. K8s is still relatively new to me.
y

Yee

09/16/2022, 9:04 PM
oh i just mean the pod that is stuck
can you
kubectl get -n flytesnacks-development get pod <pod_name> -o yaml
and see if there’s a mention of imagepullsecrets
can you also use the
-o yaml
switch on the default serviceaccount
it should look like the example
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: 2015-08-07T22:02:39Z
  name: default
  namespace: default
  uid: 052fb0f4-3d50-11e5-b066-42010af0d7b6
imagePullSecrets:
- name: myregistrykey
just want to confirm that the patch worked, that nothing else reverted it after, etc.
z

Zachary Kimble

09/16/2022, 9:07 PM
So for the service account, I think it looks correct
apiVersion: v1
imagePullSecrets:
- name: acr-pull-credentials
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-09-16T14:31:27Z"
  name: default
  namespace: flytesnacks-development
  resourceVersion: "5798"
  uid: dde5ad00-83f2-4f20-8c17-86d92f6a4001
secrets:
- name: default-token-9lvqx
And here's what I got from grepping the output from
get pod
for the stuck pod:
imagePullSecrets:
  - name: acr-pull-credentials
  nodeName: 9a8ded8a5390
y

Yee

09/16/2022, 9:14 PM
so it’s getting added… and it’s just not able to pull it. interesting.
can you just run the pod for me?
one sec let me write you the yaml
cat this to a file and try to
kubectl create -f file.yaml
apiVersion: v1
kind: Pod
metadata:
  name: debugpod
  namespace: flytesnacks-development
spec:
  serviceAccount: default
  serviceAccountName: default
  imagePullSecrets:
    - name: acr-pull-credentials
  containers:
  - args:
    - sleep
    - infinity
    image: <http://blah.azurecr.io/databricks_workflow:latest|blah.azurecr.io/databricks_workflow:latest>
    imagePullPolicy: IfNotPresent
    name: abc
i dunno if i got that right… hope so
z

Zachary Kimble

09/16/2022, 9:36 PM
It appears to have landed in the same status, ImagePullBackOff
Apparently that status counts against resource limits. I had to delete the previous ones to start
debugpod
I'm going to check the credentials again. I think that my docker config might have been updated by the az cli between when I pasted in the credentials and when I ran
docker pull
. Maybe they are actually misformatted.
Nope, confirmed that decoding the data from the secret and placing it in docker config.json works
y

Yee

09/16/2022, 9:50 PM
sorry was afk… were you able to run the debugpod?
i’m just trying to isolate out flyte
z

Zachary Kimble

09/16/2022, 9:54 PM
It doesn't run. It gets the same authentication error
y

Yee

09/16/2022, 9:55 PM
ah okay
that’s good though.
did you follow this format for the secret?
kubectl create secret docker-registry <secret-name> \
    --namespace <namespace> \
    --docker-server=<container-registry-name>.<http://azurecr.io|azurecr.io> \
    --docker-username=<service-principal-ID> \
    --docker-password=<service-principal-password>
z

Zachary Kimble

09/16/2022, 9:58 PM
No, I created the following yaml and then applied to the namespace:
apiVersion: v1
data:
  .dockerconfigjson: ***Base64 encoded json***
kind: Secret
metadata:
  name: acr-pull-credentials
  namespace: flyte-development
type: <http://kubernetes.io/dockerconfigjson|kubernetes.io/dockerconfigjson>
y

Yee

09/16/2022, 9:59 PM
what what were the keys of your json?
z

Zachary Kimble

09/16/2022, 10:00 PM
auths
   <http://address.acr.io|address.acr.io>
       auth
       identitytoken
y

Yee

09/16/2022, 10:01 PM
can you try the other way?
z

Zachary Kimble

09/16/2022, 10:03 PM
I'll try. I'm using my own AD right now rather than a SP
y

Yee

09/16/2022, 10:03 PM
sorry what’s an AD?
(I’ve never used azure before sorry, only aws and gcp)
z

Zachary Kimble

09/16/2022, 10:05 PM
active directory. It's an MS single sign on/LDAP kind of thing
It complains about the username not being a service principal GUI. I'll see if I can set up a SP and try again
Thanks for your help! At least now I know it's somewhere in the secret itself
y

Yee

09/16/2022, 10:19 PM
any azure users here know how to get image pull secrets set up?
s

Samhita Alla

09/20/2022, 5:05 AM
@Nick Müller (MorpheusXAUT), would you mind helping @Zachary Kimble? I presume that you’re on Azure.
n

Nick Müller (MorpheusXAUT)

09/20/2022, 5:36 AM
We are, unfortunately I'm out of office/on holiday until next week 😕 We were seeing similar issues, I remember we set the pull secrets on the pods directly instead of using service accounts, but I would have to look up the rest next week when I'm back, sorry
s

Samhita Alla

09/20/2022, 6:01 AM
No problem, thank you!
z

Zachary Kimble

09/21/2022, 11:58 PM
No worries Nick, Thanks for offering to look when you're back!
n

Nick Müller (MorpheusXAUT)

09/26/2022, 8:30 AM
@Zachary Kimble were you able to fix it yourself in the meantime or should I take another look at our setup? 🙂
z

Zachary Kimble

09/26/2022, 1:07 PM
@Nick Müller (MorpheusXAUT) Still stuck. Any more guidance would be appreciated. I haven't tried mounting the secret to the pod's file system. Is that what you mean by "set the pull secrets on the pods directly".
n

Nick Müller (MorpheusXAUT)

09/26/2022, 1:53 PM
sure thing, will get back to you tomorrow 🙂
@Zachary Kimble just checked our configuration. it looks like we're using something pretty similar to what you are doing. we've actually added secret creation to our slightly modified version of the flyte helm chart. it's being created as:
{{- with .Values.createImagePullSecrets }}
{{- range $secretName, $secret := . }}
---
apiVersion: v1
kind: Secret
type: <http://kubernetes.io/dockerconfigjson|kubernetes.io/dockerconfigjson>
metadata:
  name: {{ $secretName }}
  {{- if $secret.annotations }}
  annotations: {{- toYaml $secret.annotations | nindent 4 }}
  {{- end }}
  {{- if $secret.labels }}
  labels: {{- toYaml $secret.labels | nindent 4 }}
  {{- end }}
data:
  .dockerconfigjson: {{ template "imagePullSecret" $secret }}
{{- end }}
{{- end }}
where
template "imagePullSecret"
would be:
{{- define "imagePullSecret" }}
{{- printf "{\"auths\":{\"%s\":{\"username\":\"%s\",\"password\":\"%s\",\"auth\":\"%s\"}}}" .registry .username .password (printf "%s:%s" .username .password | b64enc) | b64enc }}
{{- end }}
that results in a similar secret than yours being created:
apiVersion: v1
kind: Secret
type: <http://kubernetes.io/dockerconfigjson|kubernetes.io/dockerconfigjson>
metadata:
  name: azure-acr-bshrkmain-access
data:
  .dockerconfigjson: <base64>
when decoding the
.dockerconfigjson
value, it'll have the following structure:
{
  "auths": {
    "<REGISTRY>": {
      "username": "<USERNAME>",
      "password": "<PASSWORD>",
      "auth": "<BASE64 OF USERNAME:HASH>"
    }
  }
}
The
default
service account in each flyte namespace has said image pull secret attached:
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: <NAMESPACE>
secrets:
  - name: <DEFAULT TOKEN>
imagePullSecrets:
  - name: <IMAGE PULL SECRET NAME>
I'm honestly not sure how the image pull secret is assigned the default service accounts in the flyte namespaces, didn't find any template for the cluster resource sync for it 🤔 I guess it might happen automatically/implicitly due to secrets assigned to the other flyte service accounts?
z

Zachary Kimble

09/29/2022, 10:05 PM
Thanks for the detailed configs, @Nick Müller (MorpheusXAUT). The main difference that stands out to me is "username" and "password" in the dockerconfig. I'm providing "identitytoken", so looks like that's my problem. Perhaps either k3s or flyte doesn't support logging into a private registry with an identity token. I'm going to see if I can get a service principal set up and see if that fixes things.
n

Nick Müller (MorpheusXAUT)

09/30/2022, 6:53 AM
hmm, that's a good question, we've always been using username/password for that. might definitely be worth checking out. just out of curiosity: can you manually create pods with secrets from your private registry if you're using the
identitytoken
?
z

Zachary Kimble

10/03/2022, 6:31 PM
Got it working using a service principal username/password. Thanks for the help @Nick Müller (MorpheusXAUT) And no, I wasn't able to successfully start a pod using the
identitytoken
. I get the same authorization message. So not a Flyte issue. I am able to pull the same image using docker inside the local sandbox container. My guess is that
identitytoken
doesn't work with k3s, but not sure.
n

Nick Müller (MorpheusXAUT)

10/04/2022, 12:33 PM
glad to hear you got it sorted out, although it's weird the
identitytoken
wouldn't work...