https://flyte.org logo
Join Slack
Powered by
just trying to work out how to debug this?
# flyte-deployment
r
just trying to work out how to debug this?
a
Hey @rich-painting-81385 and how does your cluster_resources section look like? That's where you'd define the value for 
defaultIamRole
for each project-domain
r
Copy code
cluster_resources:
      customData:
      - production:
        - defaultIamRole:
            value: arn:aws:iam::<account>:role/FlyteWorkers
      - staging:
        - defaultIamRole:
            value: arn:aws:iam::<account>:role/FlyteWorkers
      - development:
        - defaultIamRole:
            value: arn:aws:iam::<account>:role/FlyteWorkers
there’s not actually any annotations object showing up at all on the service account though
I’m trying this as well with a label instead of the annotation
Copy code
apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: default
      namespace: '{{ namespace }}'
      labels:
        foo: baz
but these aren’t showing up either
Copy code
#kc get sa -n nscheck3-development  default -o yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2024-08-21T15:46:32Z"
  name: default
  namespace: nscheck3-development
  resourceVersion: "300961693"
  uid: af9f4f6f-73f2-4c9e-9014-fc2486bd28f7
so it seems like this template isn’t actually being used?
I’ve tested the template for the namespace by adding labels to that and those are there on new namespaces created for projects
a
maybe there's a missing perm on the clusterRole?
at least according to the base template, the clusterRole admits CRUD ops on Service Accounts https://github.com/flyteorg/flyte/blob/7866c319d00734f10732fd493d60512500b90317/charts/flyte-binary/templates/clusterrole.yaml#L28-L34 Any interesting log coming from the flyte-binary Pod?
r
looking at the logs, the only error I can find is this, which feels fishy but it isn’t clear to me whether it’s connected or not
Copy code
flyte-binary-7c9598cf97-zpdrq flyte ERROR 2024/08/21 16:03:59 Could not cast sv to map[string]interface{}; key=%!s(MISSING), st=%!v(MISSING), tt=%!v(MISSING), sv=%!v(MISSING), tv=%!v(MISSING) default-for-task-types=[]interface {} map[string]interface {}=[map[container:container] map[container_array:K8S-ARRAY]] map[container:container container_array:k8s-array sidecar:sidecar]=<nil>
I don’t think the issue is missing permissions in the cluster role as the 
default
service account is created in the new namespaces, so the flyte binary service account has the correct permissions to do that
oh hang on flyte isn’t creating the default service account is it, that’s created automatically by k8s when the namespace gets created 🤦‍♂️
yeah, serviceaccount is missing from the resources list in the clusterrole. looking at the chart templates in github it seems that’s been added in the last month so wasn’t in v1.13
right, updated that and tested again and seems to be working! thankyou for your help @average-finland-92144
a
Cool Are you still getting that error on the binary Pod? It looks like it's complaining for a missing key on the 
default-for-task-types
map
r
Hey @average-finland-92144 apologies I missed you replying here. I’ve had a bit of a look but it’s not actually clear to me what this configuration is setting up or what format it should be in. I can’t find anything explicit in the docs so am I missing something?
a
@rich-painting-81385 the 
default-for-task-types
it's a section that should match what you have enabled in 
enabled-plugins
. the config depends on what you plan to use could you share this section of your particular values file? https://github.com/flyteorg/flyte/blob/61e75da759bbbdaa0cfd3ccbff0cd495fcfc5910/charts/flyte-binary/eks-starter.yaml#L65-L75
r
I’ve been giving this a bit more of a poke trying to understand what’s going on with it, but still getting an error
Copy code
tasks:
      task-plugins:
        default-for-task-types:
        - container: container
        - container_array: K8S-ARRAY
        enabled-plugins:
        - container
        - sidecar
        - K8S-ARRAY
        - agent-service
ERROR 2024/09/04 08:39:33 Could not cast sv to map[string]interface{}; key=%!s(MISSING), st=%!v(MISSING), tt=%!v(MISSING), sv=%!v(MISSING), tv=%!v(MISSING) default-for-task-types=[]interface {} map[string]interface {}=[map[container:container] map[container_array:K8S-ARRAY]] map[container:container]=<nil>
is this going to be because there’s not 
sidecar
and 
agent-service
entries in the 
default-for-task-types
section?
also, is there any documentation that explains what these plugins are actually used for?
hhmmm, looking more at the configmap in the cluster, it seems that this section is actually duplicated for some reason. I think I need to do a bit of tidying and have another look at the overall setup
a
got it I guess one part of it has to do with the duplicate blocks in the configmap. Agree that the docs are not explanatory about all these plugins (if you could file an Issue about that, it'd be helpful) 
agent-service
is needed if you're using any of the available Flyte Agents and 
sidecar
I think is used with Flyte copilot in sidecar mode. AFAICT if you set something in 
default-for-task-types
it should be present under 
enabled-plugins
, the other way is not required
r
Hey David, I’ve gone through the config and gotten things more in order and this error is no longer appearing. seems likely it was down to the duplicated block
13 Views
Open in Slack
PreviousNext
Powered by