https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Alex Lyashok

01/11/2024, 6:52 PM
secret was created
y

Yee

01/11/2024, 7:33 PM
could you paste the full error message please? does it come with a file/line number?
a

Alex Lyashok

01/11/2024, 9:14 PM
the message is in gen-admin-auth-secret (init) logs on flyte-backend-flyte-binary pod
Copy code
time="2024-01-11T21:12:29Z" level=info msg="Using config file: [/etc/flyte/config.d/000-core.yaml /etc/flyte/config.d/001-plugins.yaml /etc/flyte/config.d/002-database.yaml /etc/flyte/config.d/003-storage.yaml /etc/flyte/config.d/004-auth.yaml /etc/flyte/config.d/012-database-secrets.yaml /etc/flyte/config.d/014-auth-secrets.yaml /etc/flyte/config.d/100-inline-config.yaml]"
ERROR 2024/01/11 21:12:29 Could not cast sv to map[string]interface{}; key=%!s(MISSING), st=%!v(MISSING), tt=%!v(MISSING), sv=%!v(MISSING), tv=%!v(MISSING) default-for-task-types=[]interface {} map[string]interface {}=[map[container:container] map[container_array:K8S-ARRAY] map[snowflake:snowflake]] map[container:container container_array:k8s-array sidecar:sidecar]=<nil>
time="2024-01-11T21:12:29Z" level=info msg="Using config file: [/etc/flyte/config.d/000-core.yaml /etc/flyte/config.d/001-plugins.yaml /etc/flyte/config.d/002-database.yaml /etc/flyte/config.d/003-storage.yaml /etc/flyte/config.d/004-auth.yaml /etc/flyte/config.d/012-database-secrets.yaml /etc/flyte/config.d/014-auth-secrets.yaml /etc/flyte/config.d/100-inline-config.yaml]"
ERROR 2024/01/11 21:12:29 Could not cast sv to map[string]interface{}; key=%!s(MISSING), st=%!v(MISSING), tt=%!v(MISSING), sv=%!v(MISSING), tv=%!v(MISSING) default-for-task-types=[]interface {} map[string]interface {}=[map[container:container] map[container_array:K8S-ARRAY] map[snowflake:snowflake]] map[container:container container_array:k8s-array sidecar:sidecar]=<nil>
y

Yee

01/11/2024, 10:13 PM
can you remove the
-
in the
default-for-task-types
map?
it’s
map[string]string
in the code. i think some of the documentation is incorrect.
a

Alex Lyashok

01/11/2024, 10:18 PM
error went away but flye wont start
Normal Started 42s kubelet Started container flyte Warning Unhealthy 5s kubelet Liveness probe failed: Get "http://172.31.45.245:8088/healthcheck": dial tcp 172.31.45.2458088 connect: connection refused Warning Unhealthy 5s kubelet Readiness probe failed: Get "http://172.31.45.245:8088/healthcheck": dial tcp 172.31.45.2458088 connect: connection refused
y

Yee

01/11/2024, 10:21 PM
can you get logs on the binary container?
a

Alex Lyashok

01/11/2024, 10:22 PM
yeah
Copy code
2024/01/11 22:23:09 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/finisher_api.go:509
[0.457ms] [rows:1] SELECT count(*) FROM pg_indexes WHERE tablename = 'artifacts' AND indexname = 'artifacts_dataset_uuid_idx' AND schemaname = CURRENT_SCHEMA()
{"metrics-prefix":"flyte:","certDir":"/var/run/flyte/certs","localCert":true,"listenPort":9443,"serviceName":"flyte-backend-flyte-binary-webhook","servicePort":443,"secretName":"flyte-backend-flyte-binary-webhook-secret","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"gcpSecretManager":{"sidecarImage":"<http://gcr.io/google.com/cloudsdktool/cloud-sdk:alpine|gcr.io/google.com/cloudsdktool/cloud-sdk:alpine>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2","annotations":null}}
last lines
y

Yee

01/11/2024, 10:27 PM
it’s still borked?
can you do
kubectl -n flyte get pod
a

Alex Lyashok

01/11/2024, 10:27 PM
flyte-backend-flyte-binary-656697c75b-bd7rw 0/1 Running 7 (3m19s ago) 12m
y

Yee

01/11/2024, 10:28 PM
i think that should be fine
there were some restarts
a

Alex Lyashok

01/11/2024, 10:29 PM
not its in crashloop
Reason: Error Exit Code: 137
root@e486f538c2ef:~# kubectl -n flyte get pod NAME READY STATUS RESTARTS AGE flyte-backend-flyte-binary-656697c75b-bd7rw 0/1 CrashLoopBackOff 7 (2m21s ago) 16m
y

Yee

01/11/2024, 10:32 PM
ah
can you do
kubectl -n flyte logs flyte-backend-flyte-binary-656697c75b-bd7rw -p
a

Alex Lyashok

01/11/2024, 10:34 PM
ok - i found an issue
i had added an default-env-vars with a dash in it without escaping sreing
strange it gives no eror at all on that
image.png
this kills it
if you add quotes arounf the key - it works
y

Yee

01/11/2024, 10:36 PM
k
so all better?
a

Alex Lyashok

01/11/2024, 10:37 PM
let me tyry to re-add snowflake plugin cfg
y

Yee

01/11/2024, 10:37 PM
we could probably do a better job returning an error.
k
a

Alex Lyashok

01/11/2024, 10:37 PM
without "-"
dies again
image.png
image.png
these 3 lines (2 snowflake and 1 clientSecretsExternalSecretRef) is all i added
should i also "" the clientSecretsExternalSecretRef?
y

Yee

01/11/2024, 10:42 PM
any more insight in the logs?
a

Alex Lyashok

01/11/2024, 10:45 PM
as soon as i add this line - it wont start anymore
no errors in the logs
y

Yee

01/11/2024, 10:55 PM
check
-p
logs
there should be some
a

Alex Lyashok

01/11/2024, 10:56 PM
whats' -p logs?
image.png
tahts what i am looking at
y

Yee

01/11/2024, 11:08 PM
-p is previous, pulls logs from the crashed container
kubectl -n flyte logs podname -p
a

Alex Lyashok

01/11/2024, 11:12 PM
Copy code
[0.467ms] [rows:1] SELECT count(*) FROM pg_indexes WHERE tablename = 'artifacts' AND indexname = 'artifacts_dataset_uuid_idx' AND schemaname = CURRENT_SCHEMA()
{"metrics-prefix":"flyte:","certDir":"/var/run/flyte/certs","localCert":true,"listenPort":9443,"serviceName":"flyte-backend-flyte-binary-webhook","servicePort":443,"secretName":"flyte-backend-flyte-binary-webhook-secret","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"gcpSecretManager":{"sidecarImage":"<http://gcr.io/google.com/cloudsdktool/cloud-sdk:alpine|gcr.io/google.com/cloudsdktool/cloud-sdk:alpine>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2","annotations":null}}
same as current log
y

Yee

01/11/2024, 11:13 PM
that looks like the migration logs.
or the secrets init container
can you
describe
the pod?
we need to find the logs for the container that’s crashing.
kubectl -n flyte describe podname
a

Alex Lyashok

01/11/2024, 11:14 PM
it has 2 init containers and flyte in it....
y

Yee

01/11/2024, 11:14 PM
can you
-c containername
and look at all the containers individually?
if it’s crashing, there should be a pretty obvious log error
a

Alex Lyashok

01/11/2024, 11:17 PM
-c containername - ythats to what command?
y

Yee

01/11/2024, 11:17 PM
logs
a

Alex Lyashok

01/11/2024, 11:18 PM
same log for flyte - its just a bunch of select statements
kubectl -n flyte logs flyte-backend-flyte-binary-945c7f47b-mchdq -c flyte
these are 2 other logs
Copy code
root@e486f538c2ef:/app/copilot/flytepipe/admin# kubectl -n flyte logs flyte-backend-flyte-binary-945c7f47b-mchdq -c wait-for-db
<http://flyteadmin.cluster-c9mjsckwrx0i.us-east-2.rds.amazonaws.com:5432|flyteadmin.cluster-c9mjsckwrx0i.us-east-2.rds.amazonaws.com:5432> - accepting connections
root@e486f538c2ef:/app/copilot/flytepipe/admin# kubectl -n flyte logs flyte-backend-flyte-binary-945c7f47b-mchdq -c gen-admin-auth-secret
time="2024-01-11T23:11:13Z" level=info msg="Using config file: [/etc/flyte/config.d/000-core.yaml /etc/flyte/config.d/001-plugins.yaml /etc/flyte/config.d/002-database.yaml /etc/flyte/config.d/003-storage.yaml /etc/flyte/config.d/004-auth.yaml /etc/flyte/config.d/012-database-secrets.yaml /etc/flyte/config.d/014-auth-secrets.yaml /etc/flyte/config.d/100-inline-config.yaml]"
time="2024-01-11T23:11:14Z" level=info msg="Using config file: [/etc/flyte/config.d/000-core.yaml /etc/flyte/config.d/001-plugins.yaml /etc/flyte/config.d/002-database.yaml /etc/flyte/config.d/003-storage.yaml /etc/flyte/config.d/004-auth.yaml /etc/flyte/config.d/012-database-secrets.yaml /etc/flyte/config.d/014-auth-secrets.yaml /etc/flyte/config.d/100-inline-config.yaml]"
y

Yee

01/11/2024, 11:19 PM
add -p also?
a

Alex Lyashok

01/11/2024, 11:20 PM
init containers have no -p
logs
flyte one looks the same - bunch of SELECT statments
y

Yee

01/11/2024, 11:20 PM
are you able to ascertain which container is crashing?
a

Alex Lyashok

01/11/2024, 11:20 PM
flyte
it is strange that this bit you asked to remove "-" in
image.png
worked like this yesterday
but today only works without "-"
y

Yee

01/11/2024, 11:22 PM
yeah i’m seeing it both ways in various places but the actual object in code is definitely just a map of str->str
can you add snowflake and comment out the container array stuff?
a

Alex Lyashok

01/11/2024, 11:23 PM
like this?
image.png
y

Yee

01/11/2024, 11:23 PM
yeah
a

Alex Lyashok

01/11/2024, 11:23 PM
it works with array too
it's this line
image.png
that kilsl it
y

Yee

01/11/2024, 11:43 PM
but we can’t find any error message?
and that secret exists?
kubectl -n flyte get secret flyte-binary-client-secrets-external-secret returns something?
a

Alex Lyashok

01/12/2024, 12:49 AM
root@e486f538c2ef:/app/copilot/flytepipe# kubectl -n flyte get secret flyte-binary-client-secrets-external-secret NAME TYPE DATA AGE flyte-binary-client-secrets-external-secret Opaque 1 7s
y

Yee

01/12/2024, 4:23 PM
what is this secret for btw?
like what’s in it?
a

Alex Lyashok

01/12/2024, 4:24 PM
its a snowflake JWT-token
y

Yee

01/12/2024, 4:24 PM
i see
where did you get that string from? clientSecretsExternalSecretRef i don’t think that’s the right one
was it in the snowflake guide? i’m not seeing it
a

Alex Lyashok

01/12/2024, 4:26 PM
image.png
y

Yee

01/12/2024, 4:37 PM
you installed via helm right?
a

Alex Lyashok

01/12/2024, 4:37 PM
y
y

Yee

01/12/2024, 4:37 PM
could you try something for me please?
let’s rename that secret.
Copy code
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: snowflake-secret
  namespace: flyte
type: Opaque
stringData:
  FLYTE_SNOWFLAKE_CLIENT_TOKEN: <JWT_TOKEN>
EOF
remove the secrets bit from the config, the part that was making it crash
and in your helm values file, include these values
Copy code
configuration:
  inlineSecretRef: snowflake-secret
or if you’re planning on adding more, maybe rename
snowflake-secret
to `external-services`or something.
and helm apply
the
clientSecretsExternalSecretRef
i think is the wrong value to set, that one has to do with auth. don’t want to touch auth
a

Alex Lyashok

01/13/2024, 12:00 AM
ok - will try
y

Yee

01/13/2024, 1:41 AM
did it work?
a

Alex Lyashok

01/13/2024, 3:24 AM
actually got sidetracked on snowflake side - will try again soon
let me actually try with duymmy key
do i need to do
Copy code
configuration:
  inlineSecretRef: snowflake-secret
or if i already have - do i add it there?
Copy code
configuration:
   inline:
      secrets:
          adminOauthClientCredentials:
                enabled: true
                 clientSecret: Zt8RV
i tried like this
Copy code
configuration:
  inlineSecretRef: snowflake-secret
and it came up
i cant test snowflake right now but it is up
y

Yee

01/16/2024, 4:33 PM
let me know when you get it working?
but it sounds like the apikey/secret bit is resolved at least right? i can update documentation today to reflect.
a

Alex Lyashok

01/16/2024, 6:30 PM
yes - looks like key issue is solved
5 Views