https://flyte.org
Join Slack
One a closer inspection I saw that the service account was missing some configuration: ```❯ kubectl...
# flyte-deployment
g
One a closer inspection I saw that the service account was missing some configuration:
Copy code
❯ kubectl describe sa default -n flytesnacks-development
Name:                default
Namespace:           flytesnacks-development
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>
Copy code
kubectl -n flyte edit cm flyte-backend-flyte-binary-config
I had to do this ^ and then add a FLYTE_AWS_ACCESS_KEY_ID and FLYTE_AWS_SECRET_ACCESS_KEY
a
you should only need to annotate the SA with
Copy code
annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: "arn:aws:iam::<aws-account-id>:role/flyte-system-role"
g
Copy code
002_serviceaccount.yaml: |
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: default
        namespace: '{{ namespace }}'
        annotations:
          <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: '{{ defaultIamRole }}'
I have this in my help chart
a
sorry, that one yes
g
but still doesn't work
🙂
I've tried even hardcoding the role and not using the template
and adding a label
Copy code
labels:
    hello: world
and then redeployed helm
I even tried deleting the NS for 
flytesnacks-development
and the SA and let helm re-create them and it still didn't work
a
do you have the `cluster_resources`section configured?
g
yes I do
I followed the tutorial as close as possible
Copy code
cluster_resources:
      customData:
      - production:
        - defaultIamRole:
             value: YYY
[same for staging and development]
a
any relevant log from the Flyte pod?
otherwise, I'll try to repro this problem and see what's happening. You shouldn't need to use account keys nor manually annotate the KSA
g
awesome, thank you
just redeployed with extra logging
Copy code
ERROR 2024/06/27 22:36:18 Could not cast sv to map[string]interface{}; key=%!s(MISSING), st=%!v(MISSING), tt=%!v(MISSING), sv=%!v(MISSING), tv=%!v(MISSING) default-for-task-types=[]interface {} map[string]interface {}=[map[container:container] map[container_array:K8S-ARRAY]] map[container:container container_array:k8s-array sidecar:sidecar]=<nil>
this seems unrelated
Copy code
{"json":{"src":"controller.go:477"},"level":"debug","msg":"successfully read template config file [002_serviceaccount.yaml]","ts":"2024-06-27T22:37:19Z"}
{"json":{"src":"controller.go:329"},"level":"debug","msg":"Attempting to create resource [ServiceAccount] in cluster [] for namespace [flytesnacks-development]","ts":"2024-06-27T22:37:19Z"}
{"json":{"src":"controller.go:396"},"level":"warning","msg":"Failed to create kubernetes object from config template [002_serviceaccount.yaml] for namespace [flytesnacks-development] with err: serviceaccounts is forbidden: User \"system:serviceaccount:flyte:flyte-backend-flyte-binary\" cannot create resource \"serviceaccounts\" in API group \"\" in the namespace \"flytesnacks-development\"","ts":"2024-06-27T22:37:19Z"}
{"json":{"src":"controller.go:602"},"level":"warning","msg":"Failed to create cluster resources for namespace [flytesnacks-development] with err: Failed to create kubernetes object from config template [002_serviceaccount.yaml] for namespace [flytesnacks-development] with err: serviceaccounts is forbidden: User \"system:serviceaccount:flyte:flyte-backend-flyte-binary\" cannot create resource \"serviceaccounts\" in API group \"\" in the namespace \"flytesnacks-development\"","ts":"2024-06-27T22:37:19Z"}
{"json":{"src":"controller.go:611"},"level":"info","msg":"Completed cluster resource creation loop for namespace [flytesnacks-development] with stats: [{Created:0 Updated:0 AlreadyThere:1 Errored:1}]","ts":"2024-06-27T22:37:19Z"}
a
Oh there's a problem with the clusterrole
g
the tutorial didn't say anything about creating one or something around those lines
a
Yeah, maybe that's missing, but has worked for many others so I'll have to check where's the problem
g
much appreciated
is the helm chart expected to deal with the cluster role stuff
a
Yes
g
Copy code
❯ kubectl describe clusterrole flyte-backend-flyte-binary-cluster-role
Name:         flyte-backend-flyte-binary-cluster-role
Labels:       <http://app.kubernetes.io/instance=flyte-backend|app.kubernetes.io/instance=flyte-backend>
              <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
              <http://app.kubernetes.io/name=flyte-binary|app.kubernetes.io/name=flyte-binary>
              <http://app.kubernetes.io/version=1.16.0|app.kubernetes.io/version=1.16.0>
              <http://helm.sh/chart=flyte-binary-v1.12.0|helm.sh/chart=flyte-binary-v1.12.0>
Annotations:  <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte-backend
              <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
PolicyRule:
  Resources                                                   Non-Resource URLs  Resource Names  Verbs
  ---------                                                   -----------------  --------------  -----
  <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com>                               []                 []              [create delete deletecollection get list patch post update watch]
  pods                                                        []                 []              [create delete get list patch update watch]
  events                                                      []                 []              [create delete patch update]
  namespaces                                                  []                 []              [create get list patch update]
  resourcequotas                                              []                 []              [create get list patch update]
  secrets                                                     []                 []              [create get list patch update]
  <http://mutatingwebhookconfigurations.admissionregistration.k8s.io|mutatingwebhookconfigurations.admissionregistration.k8s.io>  []                 []              [create get list patch update]
  <http://customresourcedefinitions.apiextensions.k8s.io|customresourcedefinitions.apiextensions.k8s.io>              []                 []              [create get list]
  podtemplates                                                []                 []              [get list watch]
Copy code
kubectl edit clusterrole flyte-backend-flyte-binary-cluster-role
and added:
Copy code
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - create
  - get
  - list
  - patch
  - update
fixed the problem
I had to do a final:
Copy code
kubectl rollout restart deployment flyte-backend-flyte-binary -n flyte
a
Thanks! Mind fixing the tutorial?
g
sure thing
a
53 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by