https://flyte.org logo
Join Slack
Powered by
Hello, The flyteadmin pod has errors but it doesn’...
# ask-the-community
f
Hello, The flyteadmin pod has errors but it doesn’t tell more about the error. How could I find the details of what is wrong? kubectl describe pod flyteadmin-58c8458b56-7s99q -n flyte
Copy code
Name:                 flyteadmin-58c8458b56-7s99q
Namespace:            flyte
Priority:             0
Priority Class Name:  default
Node:                 ip-100-72-229-46.ec2.internal/100.72.229.46
Start Time:           Mon, 21 Aug 2023 18:24:49 -0700
Labels:               <http://app.kubernetes.io/instance=flyte|app.kubernetes.io/instance=flyte>
                      <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
                      <http://app.kubernetes.io/name=flyteadmin|app.kubernetes.io/name=flyteadmin>
                      <http://helm.sh/chart=flyte-core-v1.3.0-b5|helm.sh/chart=flyte-core-v1.3.0-b5>
                      pod-template-hash=58c8458b56
Annotations:          configChecksum: 31a9728f43480a506f9b4189a628bd7f2b7fc8bac45314eebfe8bc4cea96277
                      <http://kubernetes.io/psp|kubernetes.io/psp>: eks.privileged
Status:               Pending
IP:                   172.21.182.134
IPs:
  IP:           172.21.182.134
Controlled By:  ReplicaSet/flyteadmin-58c8458b56
Init Containers:
  run-migrations:
    Container ID:  <containerd://341ff56d080a1499f978737135289ccd9641517f34315338e86855d2e70c504>a
    Image:         <http://cr.flyte.org/flyteorg/flyteadmin-release:v1.3.0-b5|cr.flyte.org/flyteorg/flyteadmin-release:v1.3.0-b5>
    Image ID:      <http://cr.flyte.org/flyteorg/flyteadmin-release@sha256:6e32489b95294d0a00af2cfa5188544a064919b4c2507bea5795da8838ad7689|cr.flyte.org/flyteorg/flyteadmin-release@sha256:6e32489b95294d0a00af2cfa5188544a064919b4c2507bea5795da8838ad7689>
    Port:          <none>
    Host Port:     <none>
    Command:
      flyteadmin
      --config
      /etc/flyte/config/*.yaml
      migrate
      run
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 22 Aug 2023 10:07:36 -0700
      Finished:     Tue, 22 Aug 2023 10:07:37 -0700
    Ready:          False
    Restart Count:  189
    Environment:
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::178581358138:role/iam-role-flyte
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /etc/db from flyte-postgresql-manualcopy (rw)
      /etc/flyte/config from base-config-volume (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cr9lq (ro)
  seed-projects:
    Container ID:
    Image:         <http://cr.flyte.org/flyteorg/flyteadmin-release:v1.3.0-b5|cr.flyte.org/flyteorg/flyteadmin-release:v1.3.0-b5>
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      flyteadmin
      --config
      /etc/flyte/config/*.yaml
      migrate
      seed-projects
      flytesnacks
      flytetester
      flyteexamples
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::178581358138:role/iam-role-flyte
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /etc/db from flyte-postgresql-manualcopy (rw)
      /etc/flyte/config from base-config-volume (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cr9lq (ro)
  sync-cluster-resources:
    Container ID:
    Image:         <http://cr.flyte.org/flyteorg/flyteadmin-release:v1.3.0-b5|cr.flyte.org/flyteorg/flyteadmin-release:v1.3.0-b5>
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      flyteadmin
      --config
      /etc/flyte/config/*.yaml
      clusterresource
      sync
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::178581358138:role/iam-role-flyte
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /etc/db from flyte-postgresql-manualcopy (rw)
      /etc/flyte/clusterresource/templates from resource-templates (rw)
      /etc/flyte/config from clusters-config-volume (rw)
      /etc/secrets/ from admin-secrets (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cr9lq (ro)
  generate-secrets:
    Container ID:
    Image:         <http://cr.flyte.org/flyteorg/flyteadmin-release:v1.3.0-b5|cr.flyte.org/flyteorg/flyteadmin-release:v1.3.0-b5>
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
    Args:
      flyteadmin --config=/etc/flyte/config/*.yaml secrets init --localPath /etc/scratch/secrets && flyteadmin --config=/etc/flyte/config/*.yaml secrets create --name flyte-admin-secrets --fromPath /etc/scratch/secrets
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      POD_NAMESPACE:                flyte (v1:metadata.namespace)
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::178581358138:role/iam-role-flyte
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /etc/flyte/config from base-config-volume (rw)
      /etc/scratch from scratch (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cr9lq (ro)
Containers:
  flyteadmin:
    Container ID:
    Image:         <http://cr.flyte.org/flyteorg/flyteadmin-release:v1.3.0-b5|cr.flyte.org/flyteorg/flyteadmin-release:v1.3.0-b5>
    Image ID:
    Ports:         8088/TCP, 8089/TCP, 10254/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Command:
      flyteadmin
      --config
      /etc/flyte/config/*.yaml
      serve
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:                250m
      ephemeral-storage:  100Mi
      memory:             500Mi
    Requests:
      cpu:                10m
      ephemeral-storage:  50Mi
      memory:             50Mi
    Liveness:             exec [sh -c reply=$(curl -s -o /dev/null -w %{http_code} <http://127.0.0.1:8088/healthcheck>); if [ "$reply" -lt 200 -o "$reply" -ge 400 ]; then exit 1; fi; grpc_health_probe -addr=:8089] delay=20s timeout=1s period=5s #success=1 #failure=3
    Readiness:            exec [sh -c reply=$(curl -s -o /dev/null -w %{http_code} <http://127.0.0.1:8088/healthcheck>); if [ "$reply" -lt 200 -o "$reply" -ge 400 ]; then exit 1; fi; grpc_health_probe -addr=:8089] delay=15s timeout=1s period=10s #success=1 #failure=3
    Environment:
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::178581358138:role/iam-role-flyte
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /etc/db from flyte-postgresql-manualcopy (rw)
      /etc/flyte/config from clusters-config-volume (rw)
      /etc/secrets/ from admin-secrets (rw)
      /srv/flyte from shared-data (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cr9lq (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  flyte-postgresql-manualcopy:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  flyte-postgresql-manualcopy
    Optional:    false
  shared-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  scratch:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  base-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      flyte-admin-base-config
    Optional:  false
  clusters-config-volume:
    Type:               Projected (a volume that contains injected data from multiple sources)
    ConfigMapName:      flyte-admin-base-config
    ConfigMapOptional:  <nil>
    ConfigMapName:      flyte-admin-clusters-config
    ConfigMapOptional:  <nil>
  resource-templates:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      clusterresource-template
    Optional:  false
  admin-secrets:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  flyte-admin-secrets
    Optional:    false
  kube-api-access-cr9lq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type     Reason   Age                  From     Message
  ----     ------   ----                 ----     -------
  Warning  BackOff  4s (x4345 over 15h)  kubelet  Back-off restarting failed container
kubectl logs flyteadmin-58c8458b56-7s99q -n flyte
Copy code
Defaulted container "flyteadmin" out of: flyteadmin, run-migrations (init), seed-projects (init), sync-cluster-resources (init), generate-secrets (init)
Error from server (BadRequest): container "flyteadmin" in pod "flyteadmin-58c8458b56-7s99q" is waiting to start: PodInitializing
d
HI @Frank Shen What was the most recent change to the Flyte deployment? Also, could you do a 
kubectl logs flyteadmin-58c8458b56-7s99q -c run-migrations -n flyte
33 Views
Open in Slack
PreviousNext
Powered by